Which AI Crawlers to Allow in robots.txt, and Which to Block

Two kinds of AI crawler visit your site and only one of them can send anything back. Training crawlers pull pages into a model. Retrieval crawlers fetch a page so an engine can answer a live question and cite where the answer came from. Block the training ones if you want. Block the retrieval ones and you stop being quotable.

That is the whole decision. What follows is which crawler is which, and how to check whether your file is doing what you think it is doing.

Every user agent below was read from the operator's own published documentation, checked on 19 August 2026. Crawler names and their stated purposes change, so on a page like this the date it was checked matters as much as the list.

Which AI crawlers should a B2B site allow?

Allow every retrieval crawler. Those are the ones that fetch a page in order to answer somebody's question and put a link next to the answer. Training is a separate call, and a reasonable person can go either way on it depending on how they make money. The table gives the token, what the operator says the crawler feeds, what the operator publishes about its own robots.txt behaviour, and the line we run on orrjo.com.

Three of those allow lines are gestures rather than controls. ChatGPT-User, Perplexity-User and Meta-ExternalFetcher are user-initiated fetchers, and every one of those operators states in its own documentation that robots.txt may not apply to them. Write the line anyway. It costs nothing and it records what you intended.

Applebot-Extended is the strange one in the list, because it never fetches anything. Apple's documentation says it "is only used to determine how to use the data crawled by the Applebot user agent", so it is a flag on a haul that has already happened.

The lines, ready to paste

A single group can carry more than one user-agent line. Google's robots.txt specification puts it plainly: "You can group together rules that apply to multiple user agents by repeating user-agent lines for each crawler." That turns twenty groups into three.

# Retrieval and training crawlers, allowed
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Google-Extended
User-agent: bingbot
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
Allow: /

# Open corpora and undocumented crawlers, blocked
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Read the grouping before you paste it. It matters more than most people expect, and the next section explains why.

Does robots.txt actually stop AI crawlers?

It stops the operators who choose to be stopped. RFC 9309, the IETF standard that finally wrote the Robots Exclusion Protocol down in 2022, is unusually candid about this. "These rules are not a form of access authorization," it says, and "The Robots Exclusion Protocol is not a substitute for valid content security measures." Crawlers are asked to honour the file. Nothing in the protocol makes them.

Four things break in practice.

Operators who publish that they will ignore it. OpenAI, Perplexity and Meta all state that their user-initiated fetchers may skip robots.txt, on the reasoning that a person asked for the page. Whether or not you find that convincing, it is at least published, which is more than most.

Operators who publish nothing. ByteDance's Bytespider carries a feedback address in its user-agent string and a reference URL on a Chinese domain that does not resolve from the UK. There is no reachable documentation stating what the crawler collects or what it obeys.

Crawlers that stop identifying themselves. On 4 August 2025 Cloudflare published a finding that Perplexity was using "not only their declared user-agent, but also a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked", from IP addresses outside its published range. Cloudflare put the declared crawler at 20 to 25 million daily requests and the undeclared one at 3 to 6 million, observed across "tens of thousands of domains". A user-agent string is a claim, and a claim can be false.

Your own file saying something you did not mean. This one is self-inflicted, and of the four it is the one I see most often.

The mistake that quietly undoes half of these files

Group matching is winner takes all. Google's specification: "Only one group is valid for a particular crawler... Other groups are ignored." And more bluntly: "User agent specific groups and global groups (*) are not combined." RFC 9309 says the same thing in standards language, that a crawler falls back to the * group only "If no matching group exists".

So the moment you add a named group for GPTBot, GPTBot stops reading your wildcard group altogether. Every Disallow you left in there, the staging directory and the thank-you pages, no longer applies to it. Here is the shape of the error:

User-agent: *
Disallow: /admin/
Disallow: /thank-you/

User-agent: GPTBot
Allow: /

The author of that file believes they have allowed GPTBot everywhere except two private directories. They have allowed it everywhere, including both. The fix is to repeat the disallows inside every named group, which is why real files get long and repetitive. Ours does exactly that, and it is the least elegant part of it.

What does enforce

Two things, and neither is a text file. HTTP authentication is what RFC 9309 itself points you to when content genuinely must not be taken. Edge blocking is the other, and it only works if you verify the request rather than trusting the header, which means checking the source address against the operator's published ranges. OpenAI publishes its GPTBot ranges as machine-readable JSON at openai.com/gptbot.json. Anthropic's guidance is blunt about the alternative: blocking by IP address without that verification "may not work correctly or persistently guarantee an opt-out".

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content for training OpenAI's foundation models. OAI-SearchBot builds the index that ChatGPT's search feature reads from. They run independently, so disallowing GPTBot does not remove you from ChatGPT's answers, and disallowing OAI-SearchBot does. OpenAI adds a practical detail that saves a lot of anxious refreshing: for search results, "it can take ~24 hours from a site's robots.txt update for OpenAI's systems to adjust."

Almost every operator has now split its crawler along the same seam, which is the most useful thing to hold in your head when reading one of these files.

Google-Extended does not keep you out of AI Overviews

This is the most expensive misreading in the category, and a lot of people have blocked the wrong thing on the strength of it. Google's documentation on its AI features says: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search." Google-Extended is a different lever entirely. It governs training and grounding for Gemini Apps and the Vertex AI API, and Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Which leaves publishers with an uncomfortable trade. No token removes you from AI Overviews while leaving you in the ten blue links. The controls Google offers over how much of your page can be shown are nosnippet, data-nosnippet, max-snippet and noindex, and the first three shrink your ordinary search snippet at the same time.

Microsoft made a different choice, and made it early. In September 2023 the Bing Webmaster Blog set out a page-level route rather than a crawler-level one. Content tagged NOARCHIVE "will not be included in Bing Chat answers, not be linked to in the answers", and Microsoft "will not use the content for training Microsoft's generative AI foundation models". NOCACHE is the middle setting, where the answer can show your URL, title and snippet only. Both leave the ordinary listing alone: Bing confirms that "content with the NOCACHE tag or NOARCHIVE tag will still appear in our search results."

Should you block AI crawlers?

For most B2B companies selling something considered and expensive, no. The reason is arithmetic rather than principle.

In July 2025 Cloudflare published crawl-to-refer ratios measured across its network. For every visitor an AI platform sent to a site, Anthropic's crawlers made 38,065.7 requests, OpenAI's 1,091.4 and Perplexity's 194.8. Google, still operating mostly as a search engine, made 5.4. In the same analysis, 79 per cent of AI crawling was for training, 17 per cent for search and 3.2 per cent driven by user actions, against 72 per cent and 26 per cent a year earlier. Training is taking a larger share of the crawl while referral traffic fails to keep pace.

If your revenue comes from advertising against page views, that is a terrible trade and you should be aggressive about it. If your revenue comes from a small number of large contracts that begin with a buyer asking an engine who does this kind of work, one referral covers an enormous amount of crawling. That is why our own file allows almost everything, and it is the same reasoning behind the answer engine work we run for clients.

What nobody publishes is the number you would actually want, which is the share of sites in a given sector that disallow each named bot. Paul Calvano's August 2025 analysis of HTTP Archive data is the closest public work I could find, and it reports that "almost 21% of the top 1000 websites have rules for ChatGPT's GPTBot", while noting that sites "are not uniform in the bots that they choose to disallow". The per-bot disallow rates live inside interactive charts rather than in a citable figure. If someone quotes you a clean sector benchmark for this, they have made it up.

How do you check what the crawlers are actually being served?

Fetch your own page twice, once as a crawler and once as a browser, then compare the status code and the byte count. It takes about a minute per user agent and it settles arguments that otherwise run for weeks.

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" \
  https://example.com/your-page

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36" \
  https://example.com/your-page

We run this against orrjo.com. The last pass put twelve user agents through it: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Googlebot, bingbot, Google-Extended, DuckAssistBot and Amazonbot. Every one returned HTTP 200 with an identical 196,078 byte body. That is a stronger statement than "our robots.txt allows them", because it proves there is nothing at the edge quietly overruling the file.

It does not prove the crawler can read the page. None of the operator documents I opened for this piece states whether its crawler executes JavaScript, so the safe assumption is that it does not. Check the raw HTML rather than the rendered view, and check the pages carrying your prices and your positioning rather than the homepage. Those are the pages an engine reaches for when a buyer asks it a commercial question, which is the argument for treating SEO and AEO as one workstream instead of two.

ORRJO's own file, and the argument against it

Ours allows every named engine crawler and blocks two things. Trimmed for length here, because the same three disallow lines repeat inside every named group:

User-agent: *
Allow: /
Disallow: /quote-builder
Disallow: /quote-builder.html
Disallow: /blog-template.html

User-agent: GPTBot
Allow: /
Disallow: /quote-builder
Disallow: /quote-builder.html
Disallow: /blog-template.html

# the same block repeats for ChatGPT-User, OAI-SearchBot,
# ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot,
# Perplexity-User, Google-Extended, Applebot-Extended,
# cohere-ai and Meta-ExternalAgent

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://orrjo.com/sitemap.xml

The two blocks have different reasons behind them. Bytespider is blocked because there is no published commitment to hold anyone to, and no retrieval surface where we might be cited in return. CCBot is blocked because Common Crawl produces an open archive that anybody can download and train on, with no attribution path and no engine at the end of it that could send a buyer back.

The argument against both is decent, so I will make it properly. Common Crawl is a non-profit whose archive underpins a lot of open academic work. Blocking it is a small act of enclosure by a company that benefits from an open web, and it cuts off the datasets used to build the open models that might one day cite us. On Bytespider, blocking an undocumented crawler is a request it may simply ignore, so the line may be doing nothing at all beyond making the file feel tidy.

The redundancy is worth admitting too. User-agent: * with Allow: / already permits every crawler in that allowlist, so naming them individually changes nothing today. We keep it because it is a legible statement of intent, and because it survives the day somebody tightens the wildcard group without thinking about what that does to named bots.

One date worth putting in the diary

On 1 July 2026 Cloudflare announced that it is splitting AI traffic into three categories, Search, Agent and Training, and changing its defaults on 15 September 2026. From that date, "Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default."

Read the scope carefully. It applies to pages that display ads, which most B2B sites do not have, so the default is unlikely to move underneath you. Check anyway. The point is structural: if your site sits behind a CDN with bot management, robots.txt is no longer the only thing deciding who gets in, and the setting that overrules it lives in a dashboard nobody on the marketing team has ever logged into.

What to do this week

1. Read your own robots.txt the way a crawler would. Find every named group, then check whether the disallows you care about are repeated inside each one. If they are not, those directories are open to every bot you named.
2. Run the two curl commands against your pricing page, your main service page, one recent article and one page you deliberately keep out of search. Compare status codes and byte counts.
3. Separate the training decision from the retrieval decision and write them down as two groups in the file, rather than one vague policy in somebody's head.
4. Check the CDN, not only the file. A bot-management default beats a text file every time, and it changes without telling you.

If you would rather have someone run that across a whole estate and report back on what the engines do with the pages once they have them, that is what the audit is for. It is £750 and takes three working days. For the wider picture, we set out how SEO, AEO and GEO relate to each other, and if you are choosing an outside team we keep public reviews of GEO agencies and of UK B2B SEO agencies.

The file itself takes ten minutes. Deciding whether you would rather be trained on or quoted takes longer, and it is a commercial decision before it is a technical one.

Sources

• OpenAI, Bots and crawlers documentation
• Anthropic, Does Anthropic crawl data from the web
• Perplexity, PerplexityBot documentation
• Google, Google crawlers and user agents
• Google, AI features and your website
• Google, robots.txt specification
• Bing Webmaster Blog, 22 September 2023, new options for webmasters to control usage of their content in Bing Chat
• Apple, About Applebot
• Amazon, Amazonbot
• DuckDuckGo, DuckAssistBot
• Meta, Meta web crawlers
• Common Crawl, CCBot
• IETF, RFC 9309, Robots Exclusion Protocol
• Cloudflare, 4 August 2025, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
• Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
• Cloudflare, 1 July 2026, Your site, your rules: new AI traffic options for all customers
• Paul Calvano, 21 August 2025, AI Bots and Robots.txt