The Docket Index: who is blocking AI search crawlers
We fetched the robots.txt of 110 well-known websites and parsed each one with Docket's own crawler-rules engine. 26% of them block at least one AI search crawler — the crawlers that decide whether a site can appear in ChatGPT, Perplexity or Claude at all. Google's AI Overviews are deliberately not in that list: they follow your Googlebot rules, not Google-Extended, and treating the two as the same thing was our own largest source of false positives until August 2026. Most of these sites did not appear to mean to.
The finding: almost nobody separates training from search
AI crawlers are not one thing. GPTBot collects data to train OpenAI's models.
OAI-SearchBot builds the index ChatGPT searches when a user asks a question.
Blocking the first is a defensible business decision about your content being used as training
material. Blocking the second removes you from the answer.
Of the 37 sites in our sample that block any AI crawler, only 12 blocked training crawlers while leaving the search crawlers alone. The other 25 — 68% — hit a citation crawler too.
68% is true of this sample, which is news-heavy by construction. We later read the robots.txt of the Tranco top 10,000 and found the opposite: of 1,381 sites there blocking any AI crawler, 53.2% blocked training crawlers and left the search crawlers alone — the informed split, and the majority.
Both numbers are real. The difference between them is the finding: large sites with someone responsible for the robots.txt mostly get this right, and the conflation risk is concentrated in publishers and in whoever copied a block list from one.
Perplexity is blocked 2.3× more often than OpenAI's search crawler
PerplexityBot was disallowed by 25 sites. OAI-SearchBot
was disallowed by 11. That gap is not a rounding artefact — 14 sites block
Perplexity specifically while allowing OpenAI's search crawler through, which is a deliberate
editorial position rather than a misconfiguration.
Among them: theguardian.com, reuters.com, apnews.com, washingtonpost.com, wsj.com, ft.com, forbes.com, theatlantic.com….
Every crawler we measured
| Crawler | Operator | Purpose | Sites blocking | % |
|---|---|---|---|---|
| CCBot | Common Crawl | open corpus | 33 | 33.7% |
| ClaudeBot | Anthropic | training | 32 | 32.7% |
| Bytespider | ByteDance | training | 31 | 31.6% |
| Applebot-Extended | Apple | training | 27 | 27.6% |
| meta-externalagent | Meta | training | 27 | 27.6% |
| PerplexityBot | Perplexity | search index | 25 | 25.5% |
| GPTBot | OpenAI | training | 24 | 24.5% |
| Google-Extended | Gemini grounding | 23 | 23.5% | |
| Amazonbot | Amazon | search index | 23 | 23.5% |
| Claude-SearchBot | Anthropic | search index | 20 | 20.4% |
| Claude-User | Anthropic | live fetch | 20 | 20.4% |
| Perplexity-User | Perplexity | live fetch | 18 | 18.4% |
| ChatGPT-User | OpenAI | live fetch | 15 | 15.3% |
| OAI-SearchBot | OpenAI | search index | 11 | 11.2% |
By category
The spread is the story. News publishers have overwhelmingly decided to shut AI search out. Almost nobody else has.
| Category | Sites | Blocking AI search | % |
|---|---|---|---|
| News & media | 19 | 17 | 89.5% |
| Reference & knowledge | 14 | 6 | 42.9% |
| Ecommerce | 15 | 2 | 13.3% |
| SaaS & software | 20 | 0 | 0.0% |
| SEO & marketing | 10 | 0 | 0.0% |
| Local & service businesses | 11 | 0 | 0.0% |
| Education & nonprofit | 9 | 0 | 0.0% |
For a business that competes for customers rather than readers — a shop, a SaaS product, a local service — the practical read is that the field is wide open. Your competitors are almost certainly not blocking these crawlers, so being visible to them is not an advantage you can win by default. It is table stakes you can lose by accident.
Two things we did not expect
8 sites use Content-signal
Content-signal is a newer, Cloudflare-backed convention for stating intent
declaratively — Stack Overflow's reads
Content-signal: search=no, ai-train=no. No crawler is obliged to honour it. It is
worth watching because it separates what you permit from which user-agent
happens to be asking, which is exactly the distinction robots.txt handles badly.
4 sites block a conventional search engine
Not AI crawlers — Googlebot, Bingbot, DuckDuckBot or Applebot. In the cases we looked at
this is intentional: Reddit's robots.txt is a blanket Disallow: / with search
access negotiated commercially instead. It is a reminder that robots.txt describes policy,
not always practice.
Method
One GET to https://<host>/robots.txt per site, serialised
with a delay — a smaller footprint than one person visiting the homepage. Rules were parsed
with Docket's RFC 9309 implementation, which does longest-match resolution, *
wildcards and $ anchors. A site counts as blocking a crawler when that crawler
is disallowed from /.
Whether a response is a real robots.txt is decided by its content, not its status
code. Stack Overflow serves a genuine, restrictive robots.txt with an HTTP 418; a status-code
test would have miscounted it. Sites returning no User-agent: directive at all
(9 of them) are excluded from every percentage, because a site with no
robots.txt has no policy — counting it as "allows everything" would overstate how open the
web is.
Limits of this sample
Stated plainly, because the Index is only useful if you can judge it:
- It is not a random sample of the web. It is 110 well-known, high-traffic sites chosen across categories where an AI assistant might plausibly be asked for a recommendation. Percentages describe this population and nothing wider.
- 3 sites were unreachable at collection time and are excluded.
- robots.txt is a request, not a wall. It records what a site asks crawlers to do. Well-behaved crawlers comply; compliance is not measured here and we make no claim about it.
- This measures access, not citation. Being crawlable is necessary for appearing in AI answers. It is not sufficient — rendering, structure and entity clarity all matter, and none of them are in this dataset.
- It is a snapshot. Collected 2026-08-06. robots.txt files change; this one will be re-run and the figures will move.
Get the data
The full dataset, the collection script and the exact site list are in the repository. Re-run it yourself and you should get the same answer — that is the point of publishing the method alongside the numbers.
Docket runs this same audit against your site, plus 92 other checks, on your Mac. It tells you which crawlers you are blocking, whether it looks deliberate, and what the practical consequence is for each one. Download Docket →
Common questions
What percentage of websites block AI crawlers?
In Docket's August 2026 sample of 98 well-known sites with a robots.txt, 26% blocked at least one AI search crawler and 38% blocked at least one training crawler. News and media sites were the outlier at 94.7%; SaaS companies were at 5.0%.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data used to train OpenAI's models. OAI-SearchBot builds the index ChatGPT searches when a user asks a question. Blocking GPTBot keeps your content out of training data. Blocking OAI-SearchBot removes you from ChatGPT's answers. They are separate decisions and require separate robots.txt rules.
Should I block AI crawlers?
It depends on whether your content is the product. Publishers who sell access to writing have a clear reason to block training crawlers. A business that wants to be recommended to customers generally does not — blocking the search crawlers removes you from the answers where those recommendations happen.
Does blocking Google-Extended affect my Google rankings or AI Overviews?
Neither. Google-Extended controls Gemini model training and grounding in Gemini Apps and Vertex AI. Google documents the Googlebot directives as the control for AI features in Search, so blocking Google-Extended leaves both AI Overviews and classic ranking untouched. We had this wrong until August 2026 and counted it as a citation crawler; the figures on this page have been recomputed without it.