Docket / Fix it / CDN blocking AI crawlers

When your CDN blocks AI crawlers your robots.txt allows

Your robots.txt can welcome every AI crawler in the world and your server can still refuse them at the door. The refusal happens before robots.txt is ever read, it produces no entry in any SEO tool's report, and the only way to find it is to ask your own server the same question twice.

Why robots.txt cannot tell you

robots.txt is a file a crawler fetches and chooses to obey. A bot rule at your CDN is a decision made about the request itself — before anything is served, before the crawler has read a word of your policy. The two live in different places and cannot see each other.

So a site can publish a robots.txt that explicitly allows GPTBot and ClaudeBot, mean it sincerely, and return 403 to both. Nothing in the robots.txt file is wrong. Nothing in any report that reads robots.txt will say otherwise.

The usual cause is not malice or even a decision: managed bot protection ships with a known-good list, anything not on it gets challenged or blocked, and the AI crawlers are newer than most of those lists. Cloudflare, Fastly and AWS WAF all expose this as a per-user-agent rule, and the default posture on several of them has moved toward blocking AI crawlers unless you say otherwise.

The test, and the trap inside it

Send your homepage two requests and compare.

curl -sS -D - -o /dev/null \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" \
  https://example.com/ | head -1

curl -sS -D - -o /dev/null \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36" \
  https://example.com/ | head -1

Use a GET, not a HEAD. -D - -o /dev/null makes a real request and throws the body away. curl -I looks like the tidier way to ask for a status code and is the wrong tool here: measured on the same site on the same day, the blocked crawler got 403 on a GET and 200 on a HEAD, from the same machine a second apart. Bot rules are commonly scoped to the requests that actually serve content, so a HEAD can sail through a rule that stops every real visit. Check with the method a crawler uses.

The trap: a single 403 proves nothing. The same bot protection usually refuses curl whatever user-agent it sends. If you run only the first command, see 403 and conclude the crawler is blocked, you may simply have discovered that your CDN dislikes command-line tools.

Both requests, from the same machine, in the same minute. That is what makes it evidence:

Browser 200, crawler 403 — the rule is about the crawler. This is the finding.
Both 403 — your protection is blocking the tool, not the crawler. The answer has to come from your CDN's own logs; this test cannot reach it.
Both 200 — nothing to fix, and worth knowing rather than assuming.

Measured this way on one large developer platform in September 2026: a browser and an ordinary crawler both received 200, while seven AI crawlers received 403 — OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot and Claude-SearchBot. Its robots.txt allowed every one of them, with a single User-agent: * group disallowing only a search path and an API path. Re-tested minutes later from the same address, so it was user-agent based rather than rate limiting. The site is not named here; the shape is the point.

Which refusals actually cost you

Not all of these bots do the same job, and a block does not cost the same thing in each case.

Search and live-fetch crawlers — OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User. These decide whether you can be cited in an answer at all. Blocking them is a visibility loss, and it is the one that hurts.
Training crawlers — GPTBot, ClaudeBot and their equivalents. Whether to allow these is a real editorial decision with arguments on both sides, and a deliberate block is a legitimate position rather than a defect.
Everything else — ad-verification bots and similar visit pages you have submitted to them and have nothing to do with organic or AI-answer visibility. Blocking them costs nothing in search.

The defect is not "a bot was blocked". It is a bot being blocked in a way the site owner did not choose and cannot see, while their own robots.txt says the opposite.

Fixing it

Ask whoever runs your CDN or WAF to allow the specific user-agents you want, as an explicit rule rather than by relaxing bot protection generally. On Cloudflare, Fastly and AWS WAF this is a per-user-agent allow. Then re-run the two-request test rather than trusting the change — a rule that was added in the wrong order, or to the wrong hostname, looks identical to a rule that works until you ask the server.

If the block turns out to be deliberate, write it down somewhere the next person will find it. Most of the sites in this state are not making a choice; they inherited a default, and the second most common outcome after fixing it is doing the same work again in a year.

Where this sits in an audit

Docket checks this per crawler on your live server, by making the same comparison described above, and reports it against what your robots.txt claims — which is why it can say "your robots.txt allows these and your server refuses them" rather than either half alone. The robots.txt side of the question is fixing AI crawler access, and the wider surface is AI search visibility.

Worth pairing it with one other question, because they compound: a crawler that is let in still has to be able to read the page. If your content arrives after JavaScript runs, the crawlers that do not render see an empty page even with a 200 — the JavaScript SEO audit is how you check that, and a site with both problems is invisible twice over. The full list of what Docket looks at is what Docket checks, inside the technical audit.

Download Docket

Common questions

Why is GPTBot getting a 403 when my robots.txt allows it?

Because robots.txt and your CDN's bot rules are different systems that cannot see each other. robots.txt is a file a crawler fetches and obeys; a CDN bot rule is a decision about the request, made before anything is served. Managed bot protection ships with a known-good list, and AI crawlers are newer than most of those lists.

How do I check whether my CDN is blocking an AI crawler?

Request the same URL twice from the same machine — once with the crawler's user-agent, once with an ordinary browser's — and compare the status codes. Browser 200 with crawler 403 is the finding. Both 403 means your protection is refusing the tool rather than the crawler, and the answer has to come from your CDN's logs.

Does a 403 from curl prove the crawler is blocked?

No, and this is the most common mistake in checking it. Bot protection usually refuses curl whatever user-agent it sends, so a single 403 may only tell you that your CDN dislikes command-line tools. The comparison is the evidence, not the status code on its own.

Is blocking AI crawlers always a mistake?

No. Blocking training crawlers is a legitimate editorial decision. The defect is blocking search and live-fetch crawlers — the ones that decide whether you can be cited in an answer — without choosing to, while your robots.txt says the opposite.

Will an SEO tool that reads robots.txt find this?

No. Nothing in the robots.txt file is wrong, so nothing that reads it will report anything. Finding it requires asking your own server as each crawler and comparing the answer to an ordinary request.