Does Cloudflare block GPTBot by default?

By Matt Kerr·Published 15 September 2026·Updated 19 October 2026

The three OpenAI crawlers are not one thing

Almost every page written about this treats "AI crawler" as a single switch. OpenAI's crawler page lists four user agents, and three of them matter here. They do different jobs:

Crawler	What it is for	What blocking it costs you
GPTBot	Training	Your content is not used to train the model. Nothing about search or citation changes.
OAI-SearchBot	Search index	You stop appearing in ChatGPT's search results. This is the expensive one.
ChatGPT-User	Live fetch	A page a user asks about cannot be fetched and read back to them.

The fourth, OAI-AdsBot, only visits pages submitted as ads in ChatGPT, and OpenAI says the data it collects is not used to train its models. It does not bear on this question.

Blocking training is a legitimate editorial decision and plenty of publishers make it on purpose. Blocking search is almost never what someone meant to do. The question "does Cloudflare block GPTBot by default" matters mostly because of what the answer implies about the other two.

What Cloudflare actually does, from Cloudflare's own pages

Read on 19 October 2026, from the primary sources rather than from the summaries:

• 1 July 2025. Cloudflare became the first infrastructure provider to block AI crawlers by default where there is no permission or compensation, and every new domain is asked at sign-up whether to allow them. The choice is presented; it is not silently made.
• 15 September 2026. New domains onboarding get updated defaults: bots classified Training or Agent are blocked on pages that display ads, and Search remains allowed. Cloudflare's own post lists ChatGPT-User as an example of an Agent bot.

So the flat sentence "Cloudflare blocks AI bots by default", which is what every result on page one for this question says today, is no longer the whole answer. The default is now scoped by classification and by whether the page carries ads. Cloudflare describes these defaults as applying to new domains, so an existing domain starts from whatever its owner set. One related change does reach existing domains: from 15 September 2026, a crawler that does both Search and Training is also blocked by any setting that blocks Training, unless its owner opted out before that date.

The measurement: what sites actually do, and the mistake in it

Docket surveyed the Tranco top 10,000 in August 2026. Reading robots.txt, and separately asking each server. Two findings bear directly on this question.

First, the file is not the answer. 643 hosts (6.43%) refused a self-identifying bot outright (401, 403, 406, 429 or 503) before any robots.txt rule could apply. Of the hosts whose robots.txt explicitly permitted a fetch of /llms.txt, 4.73% were then denied it by the server anyway. A site can allow every AI crawler in the file and still be invisible to all of them.

Second, and this is the part that should change what you do: of the 425 sites blocking OAI-SearchBot, 414 also block GPTBot: 97.4%. Only 11 sites in the whole survey blocked search without also blocking training.

That overlap is the signature of a blanket decision rather than a considered one. Almost nobody sat down and chose to be absent from ChatGPT's search results while allowing training; they turned "AI crawlers" off as one switch, and search went with it. A default that distinguishes Training from Search (which is what Cloudflare now ships) is an improvement precisely because the blanket version is what people were living with.

How to find out what your own site does

The only reliable test is to ask your own server as each crawler and compare the answer to an ordinary request. Two requests, same machine, seconds apart:

• Browser 200, crawler 403. Your edge is refusing that crawler. This is the finding.
• Both 403. Your protection is refusing the tool, not the crawler, and the answer has to come from your CDN's logs instead.
• Both 200. Nothing is blocking on the user-agent string. Note the limit honestly: Cloudflare verifies bot identity by IP as well, so this does not prove a verified crawler gets through.

Docket does this as part of an audit: it asks your origin once per crawler using each crawler's documented user-agent, with its own name appended so anyone reading their logs can see who asked, and compares the status against an ordinary request. Nothing that only reads robots.txt can find this, because nothing in robots.txt is wrong.

What to do about it

• Decide per crawler, not per category. Blocking GPTBot and allowing OAI-SearchBot and ChatGPT-User is a coherent position: no training, still citable. A single "block AI" toggle does not express it.
• Check the edge, not the file. Your robots.txt is a request. Your server is the answer, and 6.43% of the top 10,000 sites answer differently from their file.
• Re-check after any CDN change. Managed bot rules ship with lists that update without you, and AI crawlers are newer than most of those lists.

If your crawlers are already getting a 403 while robots.txt allows them, the diagnosis and fix are on when your CDN blocks AI crawlers your robots.txt allows. The directive side (which tokens are live, which are retired and which do nothing) is AI crawler access, and the free AI crawler checker reads that file for one address, though it cannot see an edge refusal, which is why the two-request test above exists. A crawler you let in still has to be able to read the page, which is the JavaScript SEO audit.

Further reading

• A rate limit is not a block
• 'Could not be reached' is not a diagnosis
• When a report calls your decision a defect

Common questions

Does Cloudflare block GPTBot by default?

On a new domain onboarding from 15 September 2026, bots classified Training or Agent are blocked on pages that display ads, and GPTBot is the training crawler, so yes, there. ChatGPT-User, which Cloudflare gives as an example of an Agent bot, is blocked on those pages too. Search-classified bots such as OAI-SearchBot stay allowed. Every new domain has also been asked the question at sign-up since 1 July 2025. Cloudflare describes the new defaults as applying to new domains, so for an existing domain, look at its own setting.

Does blocking GPTBot stop me appearing in ChatGPT?

No. GPTBot is the training crawler. OAI-SearchBot builds the search index and ChatGPT-User fetches pages a user asks about. Blocking GPTBot alone keeps your content out of training while leaving you citable: that is a coherent position, and a single 'block AI' toggle does not express it.

My robots.txt allows GPTBot. Does that settle it?

No. In Docket's August 2026 survey of the Tranco top 10,000, 6.43% of hosts refused a self-identifying bot outright before any robots.txt rule could apply, and 4.73% of hosts whose robots.txt permitted /llms.txt were then denied it by the server. robots.txt is a request; the server is the answer.

How do I check whether my site blocks an AI crawler?

Request the same URL twice from the same machine, once with the crawler's user-agent and once with an ordinary browser's, and compare. Browser 200 with crawler 403 is the finding. Both 403 usually means your protection dislikes command-line tools rather than the crawler. Both 200 means nothing blocks on the user-agent string, though Cloudflare also verifies bot identity by IP.

Is blocking AI crawlers a mistake?

Blocking training crawlers is a legitimate editorial choice. Blocking search crawlers rarely is, and the survey suggests most people do it without meaning to: of the 425 sites blocking OAI-SearchBot, 414 also block GPTBot, and only 11 blocked search without blocking training. That overlap is the shape of one switch, not a decision.