Who blocks AI crawlers: the same 10,000 sites, two months apart

By Matt Kerr·Published 19 October 2026·Updated 20 October 2026

In August 2026 we read the robots.txt of the Tranco top 10,000 sites. On 19 October 2026 we read the same list again, as close to the same way as the August record allows. 19.5% of the files we could read now block at least one AI training crawler, down from 22.0%. The share blocking an AI search crawler is 10.5%, against 10.4% before. The drop in training-crawler blocking has a clear shape: 129 of the 171 sites that stopped blocking one had, in August, the pattern of Cloudflare's managed robots.txt block.

The headline numbers

Every percentage here is a share of the sites whose robots.txt we could read: 6,236 in August 2026 and 6,171 in October. Sites move in and out of that group, which is why the next section compares the sites that were readable both times.

The training crawlers are GPTBot, ClaudeBot, Applebot-Extended, Bytespider, meta-externalagent, CCBot. The search crawlers, the ones that decide whether a site can be cited in an AI answer, are OAI-SearchBot, PerplexityBot, Claude-SearchBot. Google-Extended is counted on its own, because Google documents it as the control for Gemini training and says AI features in Search follow the Googlebot rules. These are the same definitions the August survey used. A file blocks a crawler when it disallows the site root for that name, which is a statement about what the file asks and not about what any crawler did.

What changed on the same sites

5,889 sites had a readable file in both months. Comparing them with themselves leaves out the 347 that were readable only in August and the 282 readable only in October.

Most sites did not move. 5,423 of the 5,889 blocked exactly the same set of the 40 crawlers we checked in both months. 212 blocked more, 230 blocked fewer, and 24 added some and dropped others.

The search crawlers barely moved: 30 sites started blocking one and 33 stopped. The movement was in the training crawlers and in Google-Extended, and it ran one way. 171 sites stopped blocking a training crawler and 65 started. For Google-Extended the figures are 176 and 55.

Crawler by crawler

Shares here use all readable files in each month. Started and stopped use the same sites in both. CCBot is the name blocked most often in October, by 945 sites, as it was in August.

How much of this is Cloudflare's managed robots.txt

Cloudflare can put a block of lines at the top of a site's robots.txt. Its documentation, which we read on 19 October 2026 and which is dated 8 October 2026, shows the block: a Content-signal line, an Allow: / for everyone, and a Disallow: / for 8 named AI crawlers (Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, meta-externalagent), all between the comments # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content. We matched against that text.

October: the text itself

In October we kept each file's text, so we could look for those comments. 58 of the 6,171 readable files carry them. Only 1 of those matches the documented block line for line. 54 carry a longer list of crawler names, and 48 of those share the same list of 31 names. Cloudflare's page says the lines are generated from a site's Search, Agent and Training policies, which would explain a list longer than the one it prints. A further 53 files carry only the comment block that explains Content Signals, with no rule that restricts anything.

August: a pattern, not a text

The August survey kept what its parser made of each file and not the text, so there is no comment to look for. What it kept does show a pattern: a Content-signal line and a Disallow for all 8 documented crawlers. 276 readable files matched. 152 of them were the same length, 1,836 bytes, with the same 10 user-agent lines, one of which (cloudflarebrowserrenderingcrawler) is not in the documented block. Files that identical were generated, not typed, and Cloudflare's page says it creates the whole file for a site that has none. We say the August matches look like Cloudflare's block. We cannot prove it from parsed data.

Where the August pattern went

• 50 still match, readable in both months.
• 148 are readable in both months and no longer match. None of them carries the Cloudflare comments.
• 42 now refuse our bot when it asks for robots.txt.
• 36 are unreachable, answer with some other error, or serve a file too large to parse.

What it does to the headline

Of the 171 sites that stopped blocking a training crawler, 129 (75.4%) matched the August pattern. Of the 65 that started, 8 carry the October comments or match the pattern. Set those aside and training-crawler blocking went the other way: 57 sites started and 42 stopped, a net rise of 15 sites, or 0.3 percentage points of the 5,889. The same holds for Google-Extended, where 139 of the 176 that stopped matched the pattern.

What we cannot tell you

Why. Some owners will have switched the setting off. Cloudflare may have changed what it writes: its page for the setting was last updated 8 October 2026, and we did not test whether that matters. An origin server may have started answering with a robots.txt where there was none. What we can say is that the files differ, and that the difference is not a glitch of one fetch: we asked again, later the same day, for 55 of them, and 54 came back byte for byte the file we had stored.

Blocking that robots.txt cannot show

A server can say no without a rule. 643 hosts refused our bot outright in August, with a 401, 403, 406, 429 or 503 before any robots.txt rule applied, and 785 did in October. 209 sites whose file we could read in August now refuse us, and 81 went the other way. Of the 276 sites that matched the August Cloudflare pattern, 42 (15.2%) now refuse us, against 167 of the other 5,960 readable sites (2.8%).

We did not count a refusing host as blocking. A refusal says nothing about what the file says, and RFC 9309 lets a crawler that meets a client error such as 403 on robots.txt treat the site as having no rules, while a server error such as 503 tells it to assume everything is disallowed. Refusing hosts are not in the same-sites comparison above. They matter because the other way to keep AI out is the server, and that route is a CDN rule can do it with no trace in the file.

How we measured

Sample. The host list and the rank order are the August 2026 survey's, unchanged: the Tranco top 10,000, list PYG5J. We did not refresh it, so a site that has entered or left the top 10,000 since is not here. 10,000 hosts were attempted in both months.

Collection. On 19 October 2026, from one computer, we asked each host for /robots.txt once over https, and over plain http only when https could not connect at all. The user agent names this survey and links to the page that explains it. A timeout of 8 seconds, no retries, 16 different hosts at a time. Where the file let our user agent fetch /llms.txt we asked for that too, with one extra request for a path that cannot exist; those results have a page of their own and are not reported here. The run took 26 minutes. The August 2026 run took 16. The sites counted in the update at the top were read again with the same user agent and the same rules.

Reading the files. Docket's robots.txt parser was fixed in two ways after the August survey. A comment line or a blank line used to end a group, so the rules after it were dropped. And when a file had two User-agent: * groups, only the first was read. To compare like with like, every figure on this page uses the parser as it stood in August. Read with today's parser the same October files give 1,229 sites blocking a training crawler instead of 1,202, and 33 files read differently in all. The fix makes the October figures a little higher, not lower. We cannot re-read August with it, because August's text was not kept, which is why both months use the older reading.

Rebuilding the collector. The script that gathered the August data is in neither of our repositories, so we rebuilt it from the August records and the method published with them. We checked what can be checked: the code that turns raw records into figures, run on the August records, returns the August figures this site already published. That covers the arithmetic and not the collector. The nearest comparison for the collector is the mix of outcomes. 23.3% of hosts were unreachable in August and 22.0% in October; 62.4% had a readable file in August and 61.7% in October.

Where October may differ from August

We compared the two runs point by point. These are the same: the host list and its rank order, the 40 crawlers asked about (the same names), the parser, the 8-second timeout and no retries as August's published method states, and how an answer becomes a state (a 200 is readable, any other status is no robots.txt, no answer is unreachable). These differ, or may:

• The user agent. October's is explained on the bot page. August's string was not kept, so we cannot say the two were the same. A site that answers different agents differently would show it, most of all in the refusals.
• Pace and order. October asked 16 sites at a time in a shuffled order and took 26 minutes. August's pace and order were not recorded; it took 16.
• Plain http. October tried plain http when https could not connect: 318 sites, 146 of them readable. August's record does not say whether it did. Without those 146, training-crawler blocking is 19.6% instead of 19.5%.
• The file-size cutoff. October did not parse files above 262,144 bytes. August's cutoff was not kept, and lay above 246,207. 7 October files are larger than anything August parsed. Without them, training-crawler blocking is still 19.5%.
• The Python that ran it. October's main run used Python 3.9 and an older TLS library. August's records read as a newer Python's: they hold no unfollowed 308. That is why the redirects and the sites that had not connected were read again (see the update above). A site that the older library reached and the newer one would not is still counted, because sites that read fine were not read again.

What this does not tell you

• It is the top of the web, not the web. Percentages describe these 10,000 sites on these two days. A small business site is unlikely to be on the list.
• Some sites are in no percentage. In October 2,197 hosts were unreachable, 1,510 answered with no robots.txt, and 122 served a file above the size limit.
• A readable file is not always a real one. A 200 response that turns out to be an error page or an empty body counts as readable and blocks nothing, as it did in August.
• August's Cloudflare evidence is a pattern. Only October has the text.
• The size limit was reconstructed. The largest readable file was 246,207 bytes in August and 260,088 in October.
• One vantage point. One computer asked on one day. A site that serves different files to different visitors would not show it.
• Requests, not behaviour. Nothing here says whether any crawler obeyed any file, or whether blocking one changed anyone's traffic.

Get the data

The summary behind every table above is one small file. It holds counts and no site, and nothing on this page names one.

Download the summary (CSV)

Common questions

Did more sites block AI crawlers in October 2026 than in August?

Fewer blocked a training crawler. 19.5% of readable robots.txt files in the Tranco top 10,000 did in October, against 22.0% in August. The share blocking an AI search crawler was 10.5%, against 10.4%. On the same 5,889 sites, 171 stopped blocking a training crawler and 65 started.

What does Cloudflare's managed robots.txt look like?

A block of lines at the top of the file, between the comments # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content. It carries a Content-signal line, Allow: / for everyone, and Disallow: / for named AI crawlers. Cloudflare's page prints 8 names. In October, 58 of 6,171 readable files carried the comments.

Which AI crawler do sites block most often?

CCBot, in October, at 945 of 6,171 readable files. The search crawlers are blocked far less often than the training crawlers.

Does this page name any of the sites it measured?

No. It publishes counts and shares only, and so does the download.