For site owners

HuntrBot

Huntr is a membership for creators on Trybe: it researches the brands there so creators can find the ones worth pitching. HuntrBot is how Huntr reads those brands’ own websites.

Its User-Agent

Every request HuntrBot makes carries this User-Agent header:

Mozilla/5.0 (compatible; HuntrBot/1.0; +https://myhuntr.com/bot)

Why it reads your site

Creators pitch the brands they find on Huntr. HuntrBot reads a brand’s website so a pitch can match what the brand is selling right now (the offers it’s running, the claims it leads with, a featured product) and reach it through the contact email and social links it publishes. Huntr keeps those short notes, not copies of your pages.

What it reads, and how often

Monthly

At most once every 30 days for each brand, to see what it’s selling:

  • your robots.txt, first
  • your home page
  • one product page your home page links to

Once

When Huntr first adds a brand, to find its contact email and its Instagram and Facebook links:

  • your home page
  • if your home page shows no contact email, these pages in turn until one does:
    • /pages/contact
    • /pages/contact-us
    • /contact
    • /pages/support

To place a brand

Once for each brand, or again after 90 days if it still can’t tell, to see which country the store is in, unless your web address already says:

  • your robots.txt, first
  • /meta.json, which Shopify stores publish
  • if that doesn’t say, your home page, for its language

A page that fails here isn’t tried again.

If your home page times out, refuses or drops the connection, or answers with a 5xx server error, HuntrBot tries again up to five more times, waiting longer each time, over about 25 minutes.

If a page answers 429 Too Many Requests, HuntrBot sends nothing more to that host for as long as the answer’s Retry-After header asks, or for 60 seconds without one, an hour at most. Only then does it try again, and those tries count among the five.

Separately from brands’ sites, HuntrBot downloads the videos in Trybe’s public Top 100 from Trybe’s content network, once each, to read their first seconds of speech and the text on screen. Only that text is kept, never the video, and no brand’s site is involved.

It runs no scripts

HuntrBot fetches each page’s HTML with a plain request and reads it as text. It never runs your pages’ scripts, and it doesn’t load their images or stylesheets.

robots.txt and opting out

Before each monthly read, HuntrBot reads your robots.txt. If it disallows your site (/), the read stops there, and a product page it disallows is skipped. It follows the rules for HuntrBot, or those for * if none name it. To opt out, add this to your robots.txt:

User-agent: HuntrBot
Disallow: /

Each monthly read then stops at your robots.txt. The one-time read for contact details doesn’t check robots.txt, so this doesn’t stop it.

Questions about HuntrBot? Email hello@myhuntr.com.