Do AI crawlers read sitemaps?
We did not measure whether any AI crawler fetched a sitemap, and this page is not going to imply that we did. Our survey read robots.txt files. It can tell you what those files express. It cannot tell you what GPTBot or ClaudeBot requested next, because answering that needs your server logs, not somebody else's robots.txt.
What the standard does settle is the part most people get wrong. A
Sitemap: line is a non-group record. It sits outside the User-agent
grouping altogether, so there is no per-crawler sitemap and no way to offer one to a search engine
while withholding it from an AI crawler in the same file.
Publishing one is close to background behaviour. 62.9% of the
6,236 readable robots.txt files in our August 2026 survey carry a
Sitemap: line. Among the 1,381 sites that block at least one of the
9 AI crawlers we track, 58.4% still do — slightly fewer, not more.
What RFC 9309 actually says about sitemaps
The robots.txt standard is RFC 9309, and it deals with sitemaps in exactly one short section. Section 2.2.4, "Other Records", read at the source on 15 September 2026, says two things.
First, that a sitemap reference is not part of the protocol at all: Crawlers MAY interpret other records that are not part of the robots.txt protocol -- for example, "Sitemaps". Second, that its presence must not disturb the records that are part of the protocol: Parsing of other records MUST NOT interfere with the parsing of explicitly defined records, with the example given being that a "Sitemaps" record MUST NOT terminate a group.
Put those together and the consequence falls out of the grammar. A group is one or more
User-agent lines followed by rules, and it ends at the next User-agent
line or at the end of the file. A Sitemap: line does not end a group, and it is not a
rule inside one. No group owns it. It is addressed to the file's readers, all of them, wherever in
the file you happen to have typed it.
The RFC never uses the word "global", and we are not quoting it as though it did — that word is the consequence of the two sentences above rather than a third sentence. The consequence itself is firm, though, and it is the thing to take away: whatever your file says about who may crawl what, the sitemap it advertises is the same sitemap for everyone who reads the file.
What we measured, and the question we cannot answer
Docket read robots.txt from the Tranco top 10,000 in August 2026; 6,236 of
those files came back readable and parseable, and every figure here is over that set. For each
file we recorded whether it carried a Sitemap: line and whether it denied any of the
9 AI crawler tokens we track: 3 that feed answers and citations, and
6 that feed training corpora.
That design answers a question about expression. It does not answer the question in this page's title. To know whether GPTBot read your sitemap you would have to look at your own access log for a request from that user-agent for your sitemap URL, and no survey of other people's robots.txt files can stand in for that. We are publishing the cross-tab because nobody else has, and labelling its limit because the limit is real.
| Group of sites | Sites | With a Sitemap: line | Share |
|---|---|---|---|
| All readable robots.txt files | 6,236 | 3,921 | 62.9% |
| Blocking at least one of the 9 AI crawlers | 1,381 | 806 | 58.4% |
| Blocking a citation crawler specifically | 646 | 384 | 59.4% |
The blockers are not the outliers here
There is an obvious story to tell about those rows and it is not true, so here is the correction before the story: sites that block AI crawlers are not more likely to publish a sitemap than everybody else. They are slightly less likely — 58.4% against 62.9%, a gap of 4.5 points running the opposite way from the gotcha.
Two things are worth saying about that gap rather than leaning on it. It is small, and we are not claiming it means anything beyond its direction. And the comparison is not clean: the all-files row includes the blockers, so the true distance between sites that block and sites that block nothing is a little wider than these two percentages show. Both of those observations make the gap less interesting, which is why they are here.
The honest reading is duller and more useful than a gotcha. Publishing a sitemap is near-universal
background behaviour, and deciding to block an AI crawler barely moves it. That is the finding. A
Sitemap: line is typically written once, by whoever set the site up or by whatever
generated the file, and it is never revisited. A per-crawler Disallow block is written
later, by somebody thinking hard about one specific question, who does not scroll up. The file ends
up carrying both because nobody ever reads it top to bottom as a single statement.
A global Sitemap: line is not a leak
The second thing this data must not be read as saying: a site that blocks a crawler and
still publishes a Sitemap: line has not left a door open. If the same file
carries Disallow: / for that crawler, a crawler that complies does not request
/sitemap.xml either — the sitemap URL is under the same disallowed path as everything
else. The line is addressed to every reader; whether any given reader acts on it depends entirely on
the rest of the file.
The standing caveat applies to all of this, and it is not a small one: robots.txt binds
only the crawlers that read it and choose to comply. It is a published request, not an
access control. A crawler that ignores the file ignores the Disallow and the
Sitemap: line together, and nothing in a robots.txt survey — ours or anyone's — can
distinguish a compliant crawler from an absent one.
What to do with your own file
The decision worth making is whether a given crawler may fetch your pages. That decision lives
entirely in the User-agent and Disallow lines, and it is the only part of
the file that is per-crawler. The sitemap travels with the file, not with the group.
- If you want a crawler out, the
Disallowdoes the work. Write the group, name the token exactly as the vendor documents it, and you are done. - Deleting the
Sitemap:line achieves nothing for you and costs you something. It cannot be aimed at one crawler, so removing it removes it for the search engines you want as well — and those are the readers who actually use it for discovery. - Do not infer access from the file alone. A crawler can be allowed in robots.txt and refused by your server, which is a different failure at a different layer and needs a different test.
- Check the token, not the intent. A rule aimed at a retired or misspelled crawler name parses cleanly and applies to nobody. How often that happens is measured on its own page rather than asserted here.
Docket's audit carries an XML sitemap check in the indexability lane and separate per-crawler robots.txt and server-access checks in the AI lane, which is the split this page argues for: the sitemap is a file-level fact, access is a per-crawler one. The full list is on what Docket checks.
Method note, and what it cannot support
The has_sitemap field's collector is not in this repository. The
value arrives in the raw survey data and nothing in the published pipeline computes it, so its
meaning was checked rather than assumed: 8 live robots.txt files were fetched and
compared against the recorded value, 7 agreed and 1 was
unreachable at the time of checking. That is a spot-check of a field definition. It is not a
re-survey, and 8 files cannot confirm 6,236.
Everything else: the denominator throughout is readable, parseable robots.txt files, not the 10,000 sites attempted, and a site whose robots.txt we could not read contributes to neither the numerator nor the denominator. "Blocking" means the file expresses a denial for that token — the same definition, and the same 9 tokens, used across our directives work. No figure on this page describes a crawler's behaviour.
Related, and deliberately not repeated here: the tokens that parse but match nothing are counted in the AI directives index. Which crawler does what, and how to allow citation while refusing training, is how to fix AI crawler access. A crawler your file allows but your server refuses is when your CDN blocks AI crawlers, and the defaults that cause it are on Cloudflare and GPTBot. If you were reaching for a meta tag instead of robots.txt, start at noindex and AI crawlers. The larger question of being reachable, readable and quotable is AI search visibility.
Common questions
Do AI crawlers read sitemaps?
Our data cannot answer that, and we will not pretend otherwise. Docket's August 2026 survey read robots.txt files from the Tranco top 10,000; it records what those files express, not what any crawler requested afterwards. Answering the question for your own site means looking in your access log for a request from that crawler's user-agent for your sitemap URL. What the survey does show is that 62.9% of the 6,236 readable files publish a Sitemap: line at all.
Can I give my sitemap to Google but not to GPTBot?
No. RFC 9309 section 2.2.4 puts a Sitemap: line outside the User-agent group mechanism — it is a non-group record that must not terminate a group — so it is not scoped to any crawler. One file, one sitemap reference, offered to every reader of the file. The per-crawler decision you can make is whether that crawler may fetch your pages at all, and that lives in the Disallow lines.
Do sites that block AI crawlers still publish a Sitemap: line?
Most of them do, and slightly less often than everyone else rather than more. Of the 1,381 sites blocking at least one of the 9 AI crawlers we track, 806 publish a Sitemap: line (58.4%), against 62.9% across all 6,236 readable files. Among the 646 blocking a citation crawler specifically it is 59.4%. There is no gotcha in those numbers: publishing a sitemap is background behaviour that blocking barely moves.
Is a Sitemap: line a leak if I block a crawler?
No. If the same file carries Disallow: / for that crawler, a compliant crawler does not request your sitemap URL either, because it sits under the same disallowed path. The line is addressed to every reader of the file; whether a reader acts on it depends on the rest of the file. The standing limit applies as well — robots.txt binds only crawlers that read it and choose to comply.
Should I remove the Sitemap: line to keep AI crawlers out?
No. It cannot be aimed at one crawler, so removing it removes it for the search engines that actually use it for discovery, and it does nothing to the crawler you wanted out. The Disallow does that work. In our August 2026 survey 62.9% of readable robots.txt files carry the line, which is roughly what it looks like when a setting is written once and never revisited.