The AI rules in robots.txt that do nothing

More than half the websites that write a robots.txt rule aimed at an AI crawler are addressing something that will never read it. We read the robots.txt of the Tranco top 10,000 on 2026-08-07, parsed all 6,236 that returned one with Docket's own parser, and found that of the 1,616 sites naming at least one AI user-agent, 838 — 51.9% — name a token that no crawler uses.

These are not sites that decided to allow AI crawlers. They are sites that decided to block them, wrote the rule, and got no rule. The file parses, the syntax is valid, nothing warns, and the crawler walks past the directive because the name in it does not match its own.

Three ways a directive dies

Retired. The vendor documented the token once and has since replaced it. The old name is inert — the crawler now identifies itself as something else, so a Disallow under the old heading applies to nobody.

Anthropic's current documentation lists exactly three tokens: ClaudeBot, Claude-User and Claude-SearchBot. anthropic-ai and Claude-Web are not among them, and between them they appear on 1,024 sites in this sample. Every one of those sites believes it has made a decision about Anthropic that it has not made.

Google-NotebookLM is the live one. Google renamed it to Google-GeminiNotebook and says the old token is supported until August 2026 — this month. 67 sites in our sample name the old token, including amazon.com, pinterest.com and tiktok.com. Zero name the new one.

Undocumented. The token appears on no vendor page and in no community list. Someone wrote the company name, the product name, or a plausible-looking guess. We report these separately and Docket's own check leaves them alone, because "nobody documents it" is weaker evidence than "the vendor replaced it" — a token could be real and simply undocumented where we looked.

Cut short by the parser. The third way is the one we nearly published backwards, and the correction is more useful than the section it replaced.

RFC 9309 is exact about what a crawler name may contain. Section 2.2.1: "The product token MUST contain only uppercase and lowercase letters ('a-z' and 'A-Z'), underscores ('_'), and hyphens ('-')." No digits, no dots, no spaces. So we wrote a rule that flagged any token carrying one — which caught ChatGPT-User/2.0, written by 20 sites in this sample, and called it dead.

It is not dead. We read Google's open-source robots.txt parser rather than reasoning about it further, and RobotsMatcher::ExtractUserAgent answers the question in one line:

// Allowed characters in user-agent are [a-zA-Z_-].
while (absl::ascii_isalpha(*end) || *end == '-' || *end == '_') ++end;

That runs against the value written in your file, not only against the crawler's own string. ChatGPT-User/2.0 is therefore cut to ChatGPT-User and matches exactly what its author intended. Our rule would have told 155 sites in this sample that a working configuration was broken — the same error the whole page is about, made by us, one step from shipping.

img2dataset is the one worth staring at. It is in the community ai.robots.txt list that people copy their block rules from, and it cannot work as written: the digit is not a legal product token character, so the directive is read as img. 190 sites in this sample copied it, and bigsur.ai — also on that list, also broken, this time by the dot — accounts for another 45.

Our favourite is still 7 sites — including chatgpt.com itself — writing perplexity‑user with a U+2011 non-breaking hyphen where the ASCII one belongs. It renders identically in every editor we tried and survives copy-paste out of a styled document, so the rule reads as perplexity and nobody can see why. The others: launchpad.net, academia.edu, f95zone.to, seekingalpha.com, foreignpolicy.com, forth.gr.

One honest limit. Not every crawler runs Google's parser, and one doing naive substring matching might behave differently. That is the point rather than a caveat: a rule whose meaning depends on whose parser reads it is not a rule you can rely on.

The site that wrote the rules everyone else copies

Cloudflare authored the Content-Signal syntax we found in 384 of the 6,236 robots.txt files (6.2%). Its own robots.txt names anthropic-ai, Claude-Web and cohere-ai — all three retired — and names neither OAI-SearchBot nor Claude-SearchBot, the two crawlers that decide whether a site can be cited in ChatGPT and Claude.

That is the shape of the problem. This is not a small-site literacy gap. Keeping a list of user-agent tokens current is unglamorous maintenance that nothing prompts you to do, so nobody does it.

We were wrong about the other half of this

Our own Index reports, from a 110-site sample, that roughly two thirds of the sites blocking any AI crawler also blocked the ones that decide citation. We expected the top 10,000 to say the same thing or worse. It says the opposite.

Of the 1,381 sites here that block at least one of nine AI crawlers, 735 — 53.2% — blocked training crawlers and left the search crawlers alone. That is the deliberate, well-informed split, and it is the majority behaviour. Only 646 took the citation hit.

The asymmetry that does hold, and it is sharper

Blocking a citation crawler is almost never a standalone decision. 646 sites block at least one; when we look at OpenAI specifically, 425 sites block OAI-SearchBot and 414 of them — 97.4% — also block GPTBot. 11 sites in the entire top 10,000 block OpenAI's search crawler while allowing its training crawler. Anthropic is starker: 411 block Claude-SearchBot and 409 also block ClaudeBot, leaving 2.

Losing your place in ChatGPT's answers is, at this scale, a side effect of a training decision taken by about two sites in a thousand on purpose. That is the finding worth acting on, and it is the reason Docket separates the two crawler classes in every report instead of counting "AI bots blocked".

llms.txt: adoption is real, and the obvious check is wrong 17% of the time

We expected llms.txt adoption below 2%. It is 11.45% — 714 confirmed files across 6,236 hosts. It skews to large sites and falls steadily with rank.

Getting to a trustworthy number took two extra requests per host. A naive check — fetch /llms.txt, call a 200 a yes — returned 866 candidates. Two things were wrong with that number.

First, we fetched a control path on each host that cannot exist. 133 hosts (15.4%) answered 200 with a body for that too — catch-all handlers and soft 404s, one of them returning an image/gif for every unknown path. Among them: office.com, sentry.io, amplitude.com and dell.com.

Second, we read what came back. 13 of the survivors served something that is not an llms.txt at all: st-andrews.ac.uk returns an XML sitemap, sudoku.com and upstart.com return their robots.txt, utwente.nl returns JSON, and five hosts returned gzip bytes labelled text/plain that we could not decode and therefore did not count.

The correlation is the interesting part, and it does not point where the advocacy does. Sites with an llms.txt block a citation crawler at 1.26% against 8.3% for sites without one — a risk ratio of 0.152 (95% CI 0.08 to 0.29, on 9 events in the smaller arm, so treat the point estimate loosely). That is computed only over the 5,980 hosts we were permitted to probe, because comparing against hosts that denied us would have built the association into the sampling. Sites with an llms.txt also carry a Sitemap: directive 91.32% of the time against 60.71%.

llms.txt looks like a marker of a maintained site rather than a mechanism that does anything. Google stated in June 2026 that llms.txt files are not needed for Google Search and affect visibility neither way. Publishing one is evidence you have someone doing this job; it is not the job.

robots.txt is not access

Two separate measurements say the file and the server disagree. 643 of the 10,000 hosts (6.4%) refused a self-identifying bot outright at the edge — 401, 403, 406, 429 or 503 — before any robots.txt rule applied. And of the 5,980 hosts whose robots.txt permitted us to fetch /llms.txt, 283 (4.73%) were then denied it by the server.

A site can allow every AI crawler in robots.txt and still be invisible to all of them because a WAF rule three layers up drops unknown user-agents. No robots.txt audit — ours included — can see that from the outside. It has to be tested against the site itself, with the crawler's own user-agent.

nature.com, which gets this more right than almost anyone and still has a hole

Nature's robots.txt blocks GPTBot, PerplexityBot and ClaudeBot. We asked their server for the homepage as each of seven documented AI crawlers on 2026-08-07 and compared the answers to an ordinary browser request, which returned 200. All three of those crawlers got 406 — the file and the edge agreeing, policy enforced twice, exactly as intended.

One did not fit. Perplexity-User — the agent that fetches a page when a person asks Perplexity about it — is allowed in their robots.txt and refused 406 by their server. Nothing in the file says so. It is not a robots.txt decision at all; it is a rule in front of it, and the only way to find it is to ask.

We are naming Nature because they are among the most careful publishers we measured, not because they are careless. If a site that separates training from citation correctly, in the file, still has one crawler blocked somewhere they cannot see, the odds on a site that has never thought about it are not good.

What we did not measure, and why

We could have run those seven probes against all 10,000 hosts and published a per-crawler edge-blocking table. It would have been the most quotable thing on this page. We did not collect it.

Sending an OAI-SearchBot user-agent to ten thousand strangers to see what their servers do is unsolicited scanning, whatever the header says underneath. One request inside an audit somebody asked for is a different act. So the survey figure above is 643 hosts refusing our own self-identifying bot, which is what we were entitled to learn, and the per-crawler answer is something Docket works out for one site at a time — the site in front of it.

That is check 89, ai.edge_access. It probes the audited origin only, appends Docket-SEO-Audit to every vendor string so nobody's log shows a forged crawler, and reports the contradictions rather than the refusals: a crawler your file blocks and your server also blocks is your policy working twice, and saying so would train you to ignore the check. On Nature it reports one finding, not four.

Where someone else's data is better

Ahrefs published an llms.txt study in June 2026 across 137,210 domains — thirteen times our population — and, more importantly, with server-log request data we have no way to obtain. They found 97% of llms.txt files received no requests at all in May 2026, and that AI retrieval bots were 1.1% of requests to the ones that did. That is a stronger claim than ours about whether llms.txt is read, because it measures reading rather than presence. Our measurement covers a different question — who writes directives that cannot work — and the two agree in direction.

Method, and what it cannot tell you

Population: the Tranco top 10,000 (list PYG5J), which is a rank-aggregated list built to be harder to manipulate than a single provider's. One request for /robots.txt per host. A second request for /llms.txt only where the robots.txt we had just read permitted our own user-agent, plus one control request for a path that cannot exist. A self-identifying user-agent with a contact address, an eight-second timeout, no retries. Collected in 16 minutes on 2026-08-07.

We did not impersonate any vendor's crawler. Sending a forged OAI-SearchBot header to 10,000 third-party servers to see what they return is deception, and it would have produced better data. Docket runs the same test inside the product, against a site the person running it owns, where it is not deception.

Limits worth stating. 3,764 of the 10,000 hosts did not return a parseable robots.txt — Tranco contains CDN and infrastructure hostnames that are not websites. A robots.txt read once is a snapshot, and none of this says what any crawler actually did.

The reference list needed correcting before we could use it. "Unmatchable" is judged against the community ai.robots.txt set, and that set is itself behind vendor documentation. We read six vendor pages on 2026-08-07 and found coherebot, kagibot, meta-externalads, meta-webindexer, mistralai-index, mistralai-training, webzio all currently documented and all missing from it. Naming one of those is correct behaviour, so we excluded them rather than count them as dead. We also removed Awario's crawlers from the population — they are real, and our own pattern for "AI-intent" had swept them in, which would have inflated this page's headline by close to 300 sites.

Both corrections cut against the finding and it survives them. The direction is worth noticing anyway: the canonical list most people copy their block rules from is stale in both directions at once.

The full dataset is here — every host, its AI tokens, which of them are dead, which crawlers it blocks, and whether it has a confirmed llms.txt. Recompute it and disagree.

The check this became

Docket ships this as ai.dead_crawler_directive, and it flags a deliberately smaller number than the 51.9% at the top of this page. 782 sites — 48.4% of everyone writing an AI crawler rule — carry a token that is either vendor-retired with a documented replacement, or cut short by the parser into something no crawler is called. Those two we can prove. "Absent from the community list" we cannot, so the product does not say it.

cohere-ai, on 519 sites here, spent a day outside the check for that reason. It was obviously stale — nobody documents it — but we could not find a page saying what replaced it, and telling 519 sites their rule is dead on an assumption is the error the whole check exists to catch.

Their own sitemap had it. docs.cohere.com/docs/cohere-web-crawlers says more than we expected: Cohere's bot table reads "N/A" — they operate no crawler at all — and the blocking example they publish names Coherebot, which is the token to write if they ever do. This page previously showed cohere-training-data-crawler as the replacement, which came from a table of ours rather than from Cohere, and is corrected above. The two tables are now one file.

Severity follows the consequence rather than the tidiness. A dead heading above a Disallow means a restriction you wrote is not in force, and the crawler is reading what you meant to withhold. A dead heading above nothing but Allow costs nothing and is reported as a note.

What to do with your own robots.txt

1. Open it and list every User-agent: line that mentions an AI product.
2. Check each token against the vendor's own crawler documentation, not against a blog post or a copied gist. If it is not on the vendor's page, the rule does nothing.
3. Look for anything that is not a letter, a hyphen or an underscore. A digit, a dot, a space or an invisible character cuts the token short at that point.
4. Decide training and citation separately. They are different crawlers and different business decisions.
5. Then test access from outside the file — fetch your own homepage with the crawler's user-agent and confirm the server agrees with what you wrote.

Docket does steps two through five on every audit, names the dead tokens it finds, and prints the replacement rules to paste.

Download Docket

Common questions

What is a dead robots.txt directive?

A User-agent rule naming a token that no crawler identifies itself with — either because the vendor retired that name, or because it was never a crawler name. The file parses and nothing warns you, but the rule applies to nobody. We found one on 51.9% of the sites writing AI rules at all.

Does anthropic-ai still work in robots.txt?

No. Anthropic's current documentation lists ClaudeBot, Claude-User and Claude-SearchBot. A rule under anthropic-ai or Claude-Web matches none of them, and 563 and 461 sites respectively still use those names.

Do I need an llms.txt file?

There is no measured evidence that it helps. Google stated in June 2026 that llms.txt is not needed for Google Search and does not affect visibility either way, and Ahrefs measured 97% of llms.txt files receiving no requests at all. Adoption is 11.45% of the top 10,000 and correlates with sites that maintain their robots.txt rather than with any outcome.

Is blocking GPTBot the same as blocking ChatGPT search?

No. GPTBot collects training data; OAI-SearchBot builds the index ChatGPT searches. Of 425 sites blocking OAI-SearchBot, 414 also block GPTBot — losing citation is nearly always a side effect of a training decision rather than a decision of its own.