AI crawler directives: which to block, which to keep
Blocking GPTBot and blocking OAI-SearchBot look like the
same decision in a robots.txt file. They are not remotely the same decision, and most sites that
make one make the other by accident. Docket checks
19 AI crawlers separately, from 10 companies, because the
cost of blocking each one is different.
The distinction the whole thing turns on
An AI company points more than one crawler at your site, and they do different jobs:
- Training crawlers collect text to train a model. Blocking one means your writing does not become training data. It does not remove you from any answer.
- Search-index crawlers build the index an assistant searches when it answers a question with live citations. Blocking one removes you from those answers entirely.
- Live-fetch agents retrieve a page at the moment a user asks about it — usually because someone pasted your URL. Blocking one means the assistant cannot read a page a user explicitly asked it to read.
That is why "should I block AI crawlers?" has no single answer. Keeping your writing out of a training set while staying quotable in ChatGPT Search is a perfectly coherent position, and it is one robots.txt can express precisely — if you know which name does which job.
What we found in published robots.txt files
We read 6,236 robots.txt files. Of those, 1,616 name an AI crawler at all, and 1,381 block at least one. So far so unsurprising. The number worth sitting with is this one:
735 of those sites —
53.2% — block only training crawlers, which is the
decision described above, made deliberately and correctly. The rest did something broader.
Among sites that block OAI-SearchBot and therefore leave ChatGPT Search,
97.4% also block GPTBot.
That overlap is the tell. It is consistent with one decision — "block OpenAI" — applied to every OpenAI user-agent at once, rather than two decisions taken separately. We cannot prove intent from a file and we are not going to try: what the number shows is that the two blocks travel together almost always, and one of them has a cost the other does not.
One caveat that applies to every figure here. These are measurements of access — what a file permits — and nothing downstream. We did not measure citations, traffic or revenue, and no figure on this page should be read as measuring them.
The 19 crawlers, and what each block costs
| User-agent | Operator | Job | What blocking it does |
|---|---|---|---|
Amazonbot | Amazon | search index | Blocks Alexa and Amazon's answer surfaces. |
Applebot-Extended | Apple | training | Blocks Apple Intelligence training. Applebot (search) is separate. |
Bytespider | ByteDance | training | Blocks ByteDance/TikTok AI crawling. |
CCBot | Common Crawl | open corpus | Removes you from Common Crawl, an input to many open models. |
ChatGPT-User | OpenAI | live fetch | Stops ChatGPT fetching your page when a user asks about you. |
Claude-SearchBot | Anthropic | search index | Blocks your pages from Claude's search results. |
Claude-User | Anthropic | live fetch | Stops Claude fetching your page on a user's behalf. |
ClaudeBot | Anthropic | training | Blocks Anthropic from training on your content. |
GPTBot | OpenAI | training | Blocks OpenAI from training on your content. Does not affect whether ChatGPT can cite you live. |
Google-CloudVertexBot | grounding | Blocks Vertex AI grounding on your content for Google Cloud customers who ask it to index your site. | |
Google-Extended | training / Gemini Apps grounding | Blocks Gemini model training and grounding in Gemini Apps and Vertex AI. Does NOT affect AI Overviews, AI Mode or classic Google Search — those follow your Googlebot rules. | |
GoogleOther | research / product fetch | Blocks Google's non-Search crawls. Does not affect Search. | |
Meta-ExternalFetcher | Meta | live fetch | Stops Meta AI fetching your page on a user's behalf. |
MistralAI-Index | Mistral | search index | Removes you from Le Chat's search index and citations. |
MistralAI-Training | Mistral | training | Blocks Mistral from training on your content. |
OAI-SearchBot | OpenAI | search index | Blocks your pages from ChatGPT Search results entirely. |
Perplexity-User | Perplexity | live fetch | Stops Perplexity fetching your page on a user's behalf. |
PerplexityBot | Perplexity | search index | Removes you from Perplexity's index and citations. |
meta-externalagent | Meta | training | Blocks Meta AI training and grounding. |
The impact column is the engine's own wording, printed here from the same table the checks read, so a crawler added to Docket appears here without anyone editing this page.
Writing the rules
Directives are matched against the user-agent string, and the names are exact. A rule for
GPTBot says nothing about OAI-SearchBot, because as far as robots.txt
is concerned they are unrelated crawlers that happen to share an owner.
To stay out of training while staying quotable, name the training crawlers and leave the search ones alone:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Two mistakes are worth naming because they are silent. The first is a blanket
User-agent: * block added for some unrelated reason, which catches every crawler in
the table above including the 5 that decide whether you can be cited. The second is
blocking at the CDN or firewall instead of in robots.txt: an edge rule that returns 403 to an AI
crawler is invisible in your robots.txt file, so every tool that reads the file — including the
robots half of Docket — will tell you the crawler is allowed. Docket checks both, and reports
them separately, because a file that says yes and a server that says no is a configuration
nobody chose on purpose.
What robots.txt cannot do
It is a request, not a control. It is also the only mechanism these operators document. You
will find advice recommending noai or noimageai meta tags: none of the
operators in the table above documents support for them, so this page does not tell you they
work. If you need a guarantee rather than a request, the honest answer is authentication, not a
directive.
And robots.txt says nothing about whether an assistant can make sense of your page once it arrives. Most AI crawlers do not run JavaScript, so a page whose content is assembled in the browser is effectively blank to them even when every rule permits it — AI search visibility covers the rest of that.
Common questions
How many AI crawlers are there?
Docket checks 19 by name, operated by 10 companies, and the list changes: OpenAI split OAI-SearchBot out of GPTBot after launch and Anthropic retired Claude-Web. The number on this page is read from the product's own table rather than typed, so it cannot drift away from what Docket actually checks.
Should I block AI crawlers from my website?
It depends which one, and that is the point. Blocking a training crawler keeps your writing out of a model and costs you no visibility. Blocking a search-index crawler removes you from the answers that cite live sources. Many sites make the second decision while intending only the first.
Does blocking GPTBot stop ChatGPT citing my site?
No. GPTBot collects training data; OAI-SearchBot builds the index ChatGPT Search answers from. They are separate user-agents and a robots.txt rule for one says nothing about the other. In the robots.txt files we read, the two blocks travel together 97.4% of the time.
Do noai and noimageai meta tags work?
None of the crawler operators in Docket's table documents support for them, so this page does not tell you they work. robots.txt is the mechanism these operators publish rules for. A request is all any of them is.
Why does Docket say a crawler is blocked when my robots.txt allows it?
Because a CDN or firewall rule can return 403 to that crawler regardless of what the file says, and the file cannot see it. Docket tests access at the edge as well as reading the rules, and reports the two separately — a file that says yes and a server that says no is rarely a configuration anyone chose deliberately.