What a dead robots.txt rule actually costs

By Matt Kerr·Published 18 September 2026

Finding a retired crawler name in your robots.txt is easy, and every tool that looks will tell you. What that dead line actually costs you is a completely different question, and the answer is not in the line you are looking at — it is in the group the crawler reads instead.

Six dead names, and nothing was getting through

A computer manufacturer's robots.txt named six retired AI crawler tokens. All six genuinely retired: the vendors had replaced those names and the crawlers no longer answer to them. We reported, at medium severity, that those crawlers were reading the pages it meant to withhold from them.

That was false, and here is the file that made it false. The catch-all group carried seventeen Disallow rules. The AI group carried seven — and every one of the seven was also in the catch-all.

A crawler that matches no group falls back to User-agent: *. So those crawlers fell back to a stricter group than the one written for them, and lost no coverage whatsoever. The only directive that actually went unheard was a Crawl-delay.

The tokens were dead, the consequence was false, and the severity rested entirely on the consequence.

A rule's effect belongs to the group, not to the token

This is the part worth taking away, and it applies to every robots.txt question, not only to AI crawlers.

Writing a User-agent line does not create protection. It creates a group, and it removes that crawler from the fallback. So when the heading is a name nothing answers to, the crawler simply reads the catch-all, and three things can happen:

The catch-all group is…	What the dead name cost you
Stricter than the group you wrote	Nothing. The crawler is more restricted, not less.
Carrying the same rules	Nothing that matters. The lines are inert but the policy holds.
Looser, or missing rules the dead group had	This is the real exposure — and the only case worth a severity.

Nothing about the dead line tells you which of those you are in. The question is never "is this token current?" — it is "what does this crawler get instead?"

Sometimes the replacement is already sitting next to it

A group can be headed by several User-agent lines, and its rules apply to all of them. So a retired name stacked in the same group as its live replacement costs nothing at all: the live crawler reads that group and receives every rule in it.

That is worth knowing before you go tidying. On one site with four dead names, exactly one of them was actually uncovered — the other three sat beside live replacements or above rules the catch-all already carried. A finding that said four would have been four times the alarm and three times the wasted work.

Which way to be wrong

The comparison between the two groups is an exact match on the rule patterns, and that is deliberately crude. A catch-all group saying Disallow: / genuinely does cover a narrower rule written under a dead name — but proving that in general needs full path-matching, and a wrong answer would quietly silence a real exposure. So that case stays reported.

The direction you err in should follow what being wrong costs. A false alarm here costs somebody an afternoon reading their own robots.txt. A missed one costs the thing they were trying to protect. Those are not symmetric, so the rule is not symmetric.

Which is the opposite of how the same codebase treats what it recommends you publish →, where a confident wrong guess ends up in your markup as a claim you made. Same product, two rules, because the cost of being wrong points in two different directions.

Checking your own file

1. List every User-agent line naming an AI product, and check each token against that vendor's own crawler documentation rather than a blog post.
2. Then do the step almost nobody does: compare that group's rules against the catch-all group's. If every rule in the AI group also appears under *, a dead name there has cost you nothing.
3. Look for a live replacement in the same group. If it is there, the dead line beside it is untidy and harmless.
4. Test from outside the file. Fetch a page you meant to withhold using the crawler's user-agent string and see what the server actually does.

When a dead token really is an exposure

Keep the severity where the harm is. If your AI group disallows paths that the catch-all group does not — a members' area, a search endpoint, anything you separated out precisely because you did not want it crawled — then a retired name above those rules means they are not in force for that crawler. Nothing else in the file is protecting them. That one is worth doing today.

The fixes that make it worse

• Deleting the dead line and stopping there. You have removed a rule that did nothing and added nothing that does. Write the current token first.
• Adding the new token to a group you have not read. You are now applying that group's rules to a live crawler. Read them before you attach a working name to them.
• Treating robots.txt as access control. It is a request, honoured by the crawlers that choose to. If the content must not be fetched, that is an authentication job — the directives survey → goes into what the file can and cannot do.
• Copying an AI block from another site. That is how retired names spread in the first place: they were correct when someone published them and nothing re-checks a copied file.

How to make this finding go quiet without protecting anything

Delete the AI group entirely. There are now no dead tokens to report, and your site is governed by the catch-all alone — which is the exact situation the finding was describing. A check about dead rules cannot see rules you never wrote. The silence means the same thing it meant before; only the paperwork changed.

Where this sits in an audit

The registered check is ai.dead_crawler_directive, which reads robots.txt for crawler names that no longer match anything. For the survey of how common this is across large sites, and the three separate ways a directive dies, see the AI directives survey →. For what a noindex tag does and does not do to these crawlers, see does noindex stop AI crawlers →.

Common questions

My robots.txt names a retired AI crawler. Is my site exposed?

Not necessarily. A crawler matching no group falls back to User-agent: *, so the dead name only costs you a restriction the catch-all group does not already carry. If the catch-all is stricter, the crawler is more restricted, not less.

How do I tell whether a dead crawler rule matters?

Compare the rules in that group against the rules under User-agent: *. Anything present in the dead group and absent from the catch-all is what stopped being in force. Everything else is untidy rather than an exposure.

Does a retired token beside its live replacement do any harm?

No. A group's rules apply to every user-agent named above it, so if the current name is in the same group the live crawler receives every rule. Remove the old line when convenient.

Should I just delete my AI crawler rules if they are not working?

That removes the finding and not the problem. Deleting the group leaves your site governed by the catch-all alone, which is the situation the finding was describing. Write the current token above the rules you meant to apply.

Why would a tool report this when nothing is getting through?

Because the dead token is easy to verify and the consequence is not. Ours made exactly that mistake: it reported six retired names as an active exposure on a site whose catch-all group carried every rule the AI group had, and more.