When a report calls your decision a defect
The hard part of an audit is not finding what is wrong. It is telling a mistake from a decision — and in a configuration file those look identical. The same line means "we thought about this and chose it" on one site and "nobody has ever read this line" on another, and nothing in the file says which. Getting that wrong in the loud direction is how a tool ends up shouting at somebody for doing their job.
A published policy, reported as a critical defect
A national broadcaster names the AI crawlers that decide whether it can be cited, and disallows them. Deliberately, as editorial policy, in its own file, by people who know exactly what those names do.
We called it critical, and capped an otherwise well-run site's score over it.
Nothing was wrong. The decision had been made, by the people entitled to make it, and the report's contribution was to tell them their published policy was a defect and hand them a block of markup to undo it.
And the case that really is worth interrupting someone over
The opposite situation produces the same line in the file. A blanket
User-agent: * rule gets added for an unrelated reason — an aggressive scraper, a
staging leak, advice from years ago — and it sweeps up every crawler, including the ones that
decide whether an assistant can cite you live.
Nobody decided that. Nobody knows it happened. It is exactly the thing an audit exists to surface.
Two situations, one observation in the file: this crawler is disallowed. What separates them is not the rule. It is whether the crawler is named. A name is somebody typing a specific crawler on purpose. A wildcard is a net.
The sentence that was false about the reader's own file
Then the interesting failure, which is the reason this page exists.
The check asked whether all the blocked crawlers were named. That question has a sensible answer at each end and a dangerous one in the middle. A file that names and disallows two crawlers by hand, and lets a wildcard catch a third, answers no — not all of them are named — and the report that follows was written for a file that named none:
robots.txt blocks three AI search crawlers. None of them is named in your robots.txt, so a wildcard rule is catching them and this may never have been a decision.
Two of them were named. In that file. One line above the wildcard.
Three things make this worse than a wording slip. That sentence is the one that turns a notice into a critical, so the false claim is load-bearing. The remedy under it told the reader to re-allow two crawlers they had just deliberately blocked. And the reader can disprove the whole thing by opening the file the report is describing.
"None" is a strong word, and in a report it is usually a mis-stated "not all". When a finding makes a universal claim about your configuration, check it against the configuration — a mixed case is the one most likely to be described by a branch written for a pure one.
The fix was to stop making it choose
The set is partitioned now instead of switched on. The named ones and the wildcard-caught ones are separate findings: one records a policy at a notice, the other is the interruption. Both can appear on the same report about the same file, because both are true of different crawlers.
That is the general shape. When one sentence is asked to describe a mixed set, it will be wrong about part of it. The answer is rarely a better sentence; it is usually two findings.
The paste-ready block that ignored the measurement
One more, because it rides along with the same finding. The report offers a robots.txt snippet to paste. It used to be a fixed list — the same user-agent lines for everybody, including a crawler our own survey evidence says to leave out of that recommendation, and including crawlers the reader had not blocked in the first place.
It is derived from the measurement now: the block names the crawlers actually found to be caught, and nothing else. A snippet is something people paste, which makes it the one part of a report that must never be generic. That is a cousin of a suggested markup block carrying somebody else's facts → — there the example values read as researched; here the list simply did not match the finding above it.
Reading this on your own report
- Open the file and look for the name. If the crawler is named, someone typed it. If only a wildcard covers it, decide now whether you meant to.
- Check any universal claim against the file itself. "None", "every" and "all" are where reports go wrong, and your file is the evidence.
- Decide training and citation separately — they are different crawlers doing different jobs, and the costs are not comparable: which AI crawler does what →.
- Check the edge as well as the file. A file that says yes and a server that says no is a configuration nobody chose.
When this is a real problem
Keep the weight where it belongs. A wildcard catching search-index crawlers removes you from answers that cite live sources, and it is invisible precisely because the rule was added for something else and has been sitting there ever since. That is worth the interruption. A named block is worth a line in the report and nothing more.
The fixes that make it worse
- Undoing a deliberate block because a report shouted. If your organisation decided to stay out of those answers, a tool's severity is not a reason to reverse it. Tools do not know your policy.
- Blocking the whole set because one crawler is a problem. The names are matched exactly and the jobs are different; a rule for one says nothing about another.
- Fixing the file and leaving the firewall. An edge rule refusing a crawler is invisible in robots.txt, so every tool reading the file will tell you the crawler is allowed.
- Pasting a suggested block without reading it. Check that every line names a crawler you actually meant to change.
How to make this read as policy without deciding anything
Name every crawler explicitly in your file. The finding drops from an interruption to a note, because naming is the signal of intent — and naming costs you nothing and commits you to nothing. The check is reading a proxy for deliberateness, not deliberateness, and no file can carry the difference. Worth knowing in both directions: it is why a named block is treated gently, and why a wildcard deserves a look even when you are fairly sure you meant it.
Where this sits in an audit
The registered check is ai.crawler_access, which reads robots.txt for AI crawler
access. For what each crawler does and what blocking it costs, see
the AI crawler directives →. For a rule that names
a crawler nothing answers to, see what a dead
robots.txt rule actually costs →. For the case where the audit itself was refused, see
telling a rate limit from a block →.
Common questions
An audit says blocking AI crawlers is a critical issue. Is it?
Only if you did not mean to. A named, deliberate block is a policy and should be reported as a note; a crawler swept up by a wildcard rule added for some other reason is the case worth acting on, because nobody decided it.
How can a tool tell a deliberate block from an accidental one?
It cannot, exactly. It reads a proxy: whether the crawler is named. Naming a specific user-agent is someone typing it on purpose, while a wildcard is a net. That proxy is good enough to set severity and not good enough to be certain.
My report says none of my crawlers are named, but some are. Why?
Because a test asking whether all of them are named answers no as soon as one is not, and the sentence underneath was written for a file that named none. Check any universal claim against the file itself.
Should I re-allow crawlers my organisation deliberately blocked?
No. A severity level is not a policy decision. If the block was considered and chosen, the correct outcome is that the report records it and moves on.
Is fixing robots.txt enough to let an AI crawler in?
Not always. A rule at your CDN or firewall that refuses the crawler is invisible in robots.txt, so a file that permits it and a server that refuses it will both be true at once. Check the response your server actually gives.