What is llms.txt?
llms.txt is a Markdown file at the root of a website that lists the pages a language model should read, with a one-line note on each. It is a proposal, not a standard, and Google says you do not need it: its documentation on AI Overviews states that you don’t need “AI text files” to appear there.
Where it came from and what goes in it
Jeremy Howard proposed it on 2024-09-03 at llmstxt.org (read 2026-09-28). The format is deliberately small:
- an H1 with the site or project name, which is the only required part;
- a blockquote with a short summary;
- optional paragraphs of detail;
- H2 sections, each a list of links written as
- [Title](url): note; - by convention, a section called Optional for things a model can skip when it is short of room.
The point is a short, curated map, not a copy of the site. A model given a long page of navigation, cookie banners and scripts has to work out what matters; a model given this file is told. Ours is at docketseo.app/llms.txt, and it is generated from the same data as the pages it lists, so it cannot quote an old price.
Does it do anything?
For search, nothing anyone has shown. Google’s page on AI features and your website (updated 2025-12-10) says there are “no additional requirements to appear in AI Overviews or AI Mode” and that you “don’t need to create new machine readable files, AI text files, or markup”. Ranking and citation still come from the crawlers reading your actual pages.
Where it can help is the narrower case it was designed for: a tool or an assistant that has already decided to read your site, often a developer pointing an agent at documentation. The AI labs publish llms.txt files for their own developer docs for exactly that reason. It is cheap to make, harmless to have, and worth keeping honest: a genuine index of your best pages, not a list of keywords.
How many sites actually have one
More than you would guess. In Docket’s August 2026 survey of 6,236 sites with a readable robots.txt in the Tranco top 10,000, 11.45% had a real llms.txt (714 files), and the share is highest among the largest sites. Getting that number right took care: our first pass found 866 apparent “files”, and 146 of them (16.9%) were not one. Soft 404 pages, sitemaps, robots.txt served at the wrong path. Any tool that flags a “missing” or “present” llms.txt without reading the body, or without asking for a path that cannot exist, is wrong at least one time in six. The full breakdown is in the Docket Index.
We read the same list again on 19 October 2026, and 13.4% had a real llms.txt (828 files). The change since August is on its own page.
The file is useless if the crawler cannot fetch it
The same survey found that among sites whose robots.txt allowed a fetch of
/llms.txt, 4.73% refused it at the server: the file said yes
and the firewall said no. Before writing an llms.txt, make sure the crawlers you are writing it for
can reach your pages at all.
Check your own site
An llms.txt only matters to a crawler that is allowed in. The free AI crawler checker reads your robots.txt the way the crawlers do and shows, crawler by crawler, which ones may fetch your pages and which rule decided it.
$349once · or 4 × $87.25
For Macs with Apple silicon or Intel, macOS 12 or later
On your phone?
Founding price $174.50 once · or 4 × $43.63, first 25 buyers, then $349. Buy applies the code FOUNDINGDOCKET for you; if the checkout shows $349, enter it under ‘Have a discount code?’.
Screaming Frog is $279 a year. Ahrefs plans with Site Audit run $29–$449 a month. Docket is $349 once. Their prices read 8 October 2026.
30 days, no conditions, no questions asked. Refund policy
Related
- AI search visibility: what actually decides whether you are cited
- AI crawlers and sitemaps
- Every AI crawler: which to block and which to keep
Common questions
Does llms.txt help me rank in Google or appear in AI Overviews?
Google's documentation says there are no special requirements to appear in AI Overviews or AI Mode, and that you don't need AI text files to be included. Crawler access to your real pages is what counts.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may fetch. llms.txt is a suggested reading list for a model that is already reading your site. It grants and blocks nothing.
Should I add one anyway?
It is cheap and harmless if it is a genuine index of your best pages, generated from the same source as those pages so it cannot go stale. Check first that AI crawlers can reach your site; a file nobody can fetch does nothing.