llms.txt in the top 10,000 sites, two months apart

By Matt Kerr·Published 19 October 2026

In August 2026 we asked the Tranco top 10,000 sites for their /llms.txt. On 19 October 2026 we asked the same list again, as close to the same way as the August record allows. 828 sites had a real file in October, which is 13.4% of the 6,171 whose robots.txt we could read. In August it was 714, or 11.4%. The rise is not only a different set of sites answering: of the 5,595 sites we could ask both times, 137 started having a real file and 21 stopped.

The headline numbers

Shares here are of the sites whose robots.txt we could read, the base the August page used for its own figure. We asked for /llms.txt only where that robots.txt let our survey bot fetch it, so some readable sites sit in the base and could not be in the count: 310 in October and 256 in August. The last row leaves them out.

A file counts as real when it passes three tests, the same three the August page used:

• the request for /llms.txt answered 200;
• a made-up path on the same host did not also answer 200, because a host that does is answering yes to everything;
• the first bytes are not another kind of file: XML, JSON, an HTML comment or page, a robots.txt or sitemap, compressed data the survey could not read, or nothing.

A 200 answer is not a file

Counting every site that answered 200 would have found 1,767 in October and 1,676 in August. The tests leave 828 and 714. That is 46.9% of the 200 answers in October and 42.6% in August. Most of the rest were pages and empty responses: 765 of the October 200 answers were served as text/html and 16 had an empty body. The survey's first filter set those aside before the tests began. A check of your own site should do the same: read the first lines of what comes back, and ask for a path that cannot exist before it believes a 200.

The made-up path removes the most. 144 candidates in October sat on a host that also answered 200 for a path we invented, which is what a catch-all rule or a soft 404 page does, against 133 in August. Another 12 were a different kind of file: compressed data the survey could not decode (6), an HTML page or comment (2), a robots.txt (2), JSON (1) and XML (1). In August the figure was 13.

One note on August's arithmetic. That page sets aside 146 of its 866 candidates, so 866 less 146 is not 714. The other 6 did not answer 200 in the validation step, though the survey had recorded a 200 for them, and a file we could not confirm is not counted. October's collector read each file once and checked that response, so its row is 0.

What the servers said

This is what /llms.txt answered at the 5,861 sites we could ask in October, with August's 5,980 beside it.

In October 5.2% of the sites we could ask refused the request, against 4.7% in August. Their robots.txt said our bot could ask and their server said no, the gap the August page covers under “robots.txt is not access”. A CDN setting is one way a server can do that and leave no trace in robots.txt; we cannot tell which way these sites did it. A refused request is not counted as a missing file, because we do not know what was behind it.

What changed on the same sites

5,595 sites had a readable robots.txt that let us ask in both months. Comparing them with themselves removes the effect of which sites answered each time.

That is 116 more real files on the same sites, 16.9% more than in August. 137 sites started against 21 that stopped.

Most of the sites that stopped do not look like a deleted file. Of the 21:

• 6 now answer 404 or 410, which is what a deleted file looks like.
• 8 answer 200 for the made-up path as well, so their file now fails the second test. Whether it is still there we cannot tell.
• 3 refuse our bot.
• 2 answer 200 with something that is not a text file.
• 2 gave a server error or a redirect we could not complete.

Most of the sites that started had no file in August. Of the 137, 110 answered 404 or 410 then, 10 answered 200 for the made-up path too, 6 refused our bot, and 11 did something else.

The full counts moved by 114, and the same-site comparison explains it: 137 that started, minus 21 that stopped, minus 28 that had a file in August and could not be asked in October, plus 26 that have one in October and could not be asked in August, is 114. Of the 28 we could not ask in October, 22 refused our bot at robots.txt, 4 disallowed our bot from /llms.txt, 1 could not be reached, and 1 had no robots.txt.

Where in the list

The ranks are Tranco's and unchanged from August. The share and the count rose in every band. The top 1,000 sites were the most likely to have a file in both months and the last band the least.

What the files look like

For each file we confirmed we kept the first 300 characters and its size. 789 (95.3%) of the October files open with a Markdown H1, the one required part of the format described on what llms.txt is, against 664 (93.0%) in August. 640 (77.3%) have a blockquote summary straight after it, against 507 (71.0%). 792 were served as text/plain and 33 as text/markdown.

The middle file is 8,302 bytes, against 7,539 in August. 68 are larger than 100 KB (August: 62) and 16 are smaller than 1 KB (August: 19). We cannot say anything about the links or sections further down a file, because we did not keep them.

Sites that have a file block AI crawlers less often

The August page found that sites with an llms.txt were less likely to block an AI search crawler. October shows the same shape. These shares use the sites we could ask, and the crawler groups are the ones on the crawler-blocking page.

Only 6 of the 828 sites with a file block a search crawler, which is too few for the ratio (0.085, 95% interval 0.04 to 0.19) to be read closely. The table says which sites have both things. It does not say that publishing a file changes who gets blocked, or the reverse, and nothing here shows what links them. Training-crawler blocking fell in both groups since August, which the crawler-blocking page looks at.

How we measured

Sample and dates. The sites are the Tranco top 10,000 (list PYG5J): the same hosts in the same order as the August 2026 survey, read on 7 August 2026 and on 19 October 2026. How the run was set up, and how the robots.txt files were parsed so that both months read alike, is on the crawler-blocking page. The survey's user agent, Docket-SEO-Audit-Survey, is explained on the bot page.

The llms.txt request. After reading a site's robots.txt, we asked for /llms.txt only if that file let the survey bot fetch it. When what came back looked like a file, we asked once more, for a path that cannot exist on that host. One request each, no retries, and a timeout of 8 seconds.

Same tests, both months. The tests above are applied unchanged to both months. Before anything is counted for October they are run on the August records, and the build stops unless they give back the figures the August page published. They do.

What this does not tell you

• Whether anyone reads these files. This counts files that exist. The August page covers what is known about use.
• The top of the web, not the web. These are 10,000 popular hosts on two days. A small business site is unlikely to be on the list.
• Some sites are in no share. 310 readable sites did not let our bot ask. Others had no readable robots.txt at all, including the sites that refused our bot there, which the crawler-blocking page counts.
• Some sites were read again. The October collector ran under Python 3.9 with an older TLS library, and did not follow an HTTP 308 redirect. 108 sites answered /robots.txt with one and 10 answered /llms.txt with one; 2,576 more had not connected or had fallen back to plain http. All 2,694 were read again on 20 October 2026, under Python 3.11 with redirects followed, and every figure on this page includes the new reads: 46 of those sites have a real llms.txt. Sites that read fine on 19 October 2026 were not read again.
• Some differences from August cannot be removed. August's user agent, its pace and order, whether it fell back to plain http, and its file-size cutoff were not recorded. The crawler-blocking page lists them. Without the 146 sites read over plain http, the share of readable files with a real llms.txt would be 13.5% instead of 13.4%.
• August's first filter could not be recovered. The rule that turned a 200 into a candidate was rebuilt for October. The tests after it are identical, and only 5 of the 158 sites that started or stopped sit on the difference.
• Only how files open. We kept 300 characters of each file.
• A real file is not a good or a current one. We checked that it is there and what it opens with.
• One computer, one day each. A site that shows different visitors different files would not show it. Two months is a short gap, 21 sites stopped, and we ran no test of whether any difference is more than chance.

Get the data

The summary behind every table above is one small file. It holds counts and no site, and nothing on this page names one.

Download the summary (CSV)

Related

• What is llms.txt, and does it do anything?
• The AI rules in robots.txt that do nothing, the August survey
• AI crawler blocking, August to October 2026

Common questions

How many of the top 10,000 websites have an llms.txt?

On 19 October 2026, 828 of the 6,171 sites in the Tranco top 10,000 with a readable robots.txt had a real llms.txt, which is 13.4%. In August 2026 it was 714, or 11.4%.

Does a 200 response at /llms.txt mean a site has one?

No. 1,767 sites answered 200 in October and 828 had a real file, 46.9%. The rest were mostly web pages, empty responses or hosts that answer 200 for any address, including one that cannot exist.

Did more sites add an llms.txt between August and October 2026?

Yes, on the same sites. Of 5,595 sites we could ask both times, 137 started having a real file and 21 stopped, and only 6 of those 21 now answer the way a deleted file does.

Does this page name any of the sites it measured?

No. It publishes counts and shares only, and so does the download.