What is ClaudeBot?
ClaudeBot is Anthropic’s web crawler, and it collects pages to train Claude models. Blocking it keeps your future pages out of Anthropic’s training data; it does not remove you from Claude’s answers, which are fetched by two other crawlers, Claude-SearchBot and Claude-User.
Anthropic runs three crawlers, not one
Anthropic’s own help article (dated 2026-04-07 on the article, re-read 2026-10-19) names three, and each one answers a different question about your site:
| Crawler | What Anthropic says it does | What blocking it does |
|---|---|---|
ClaudeBot | “helps enhance the utility and safety of our generative AI models by collecting web content” | Your future pages are excluded from training data. |
Claude-SearchBot | “navigates the web to improve search result quality for users” | Your pages are not indexed for Claude’s search. |
Claude-User | fetches a page “when individuals ask questions to Claude” | Claude cannot open your page when someone asks it about you. |
So the question to ask is not “should I block Claude?” but which of three things
you want to stop. Refusing training while staying findable is a coherent position: block
ClaudeBot, leave the other two open.
How to block ClaudeBot, and only ClaudeBot
Two lines in the robots.txt at the root of your site:
User-agent: ClaudeBot
Disallow: /Anthropic says its crawlers honour robots.txt, and that they also support
Crawl-delay, the non-standard line that asks a crawler to slow down rather than stay
away. If the load is the problem rather than the use, a delay is the gentler answer.
A group that names ClaudeBot replaces your User-agent: * rules for it
completely, so anything you disallow for everyone has to be repeated inside the named group if it
should still apply. And a rule written with a stray character (a version number, a
non-breaking hyphen pasted from a document) can address a name no crawler has; the
Docket Index found such rules on real sites.
What most sites that block Anthropic actually do
In Docket’s August 2026 survey of the Tranco top 10,000, 411 sites block Claude-SearchBot, and 409 of them block ClaudeBot as well, leaving 2 that give up Claude’s search while still allowing training. The pattern is the same at OpenAI: sites that close the search crawler almost always close the training crawler too, which suggests most of those search blocks were a side effect of blocking “AI” as one thing rather than a decision to disappear from answers.
Is a request really from ClaudeBot?
Anyone can put “ClaudeBot” in a user-agent string. Anthropic publishes the addresses its crawlers use at claude.com/crawling/bots.json, so a request can be checked against that list before you block or allow it at your server.
What this site’s own robots.txt says
Docket’s own file, read when this page was built, names ClaudeBot and
Claude-SearchBot separately and gives each Allow: /. It does not name
Claude-User, so the User-agent: * group applies to it, and that group allows
everything too. All three Anthropic crawlers may fetch this site, and the file keeps the search
crawlers and the training crawlers in separate groups so the next person to edit it can see the difference.
robots.txt is a request, not a lock
Everything above is about what your file asks. Your CDN or firewall can refuse a crawler the file welcomes, and a bot-protection setting switched on by someone else is the usual reason a site that meant to stay open is not. That is also why the checker below reads the file and says plainly what it cannot see.
Check your own site
Most sites have never looked at what their file says to Anthropic’s three crawlers. The free AI crawler checker reads your robots.txt the way the crawlers do and shows, crawler by crawler, which ones may fetch your pages and which rule decided it.
$349once · or 4 × $87.25
For Macs with Apple silicon or Intel, macOS 12 or later
On your phone?
Founding price $174.50 once · or 4 × $43.63, first 25 buyers, then $349. Buy applies the code FOUNDINGDOCKET for you; if the checkout shows $349, enter it under ‘Have a discount code?’.
Screaming Frog is $279 a year. Ahrefs plans with Site Audit run $29–$449 a month. Docket is $349 once. Their prices read 8 October 2026.
30 days, no conditions, no questions asked. Refund policy
Related
- Every AI crawler: which to block and which to keep
- Does noindex stop AI crawlers?
- When your CDN blocks crawlers your robots.txt allows
Common questions
Does blocking ClaudeBot remove my site from Claude?
No. ClaudeBot collects training data. Claude's answers come from Claude-SearchBot, which indexes pages for search, and Claude-User, which fetches a page when someone asks Claude about it. Block ClaudeBot alone and you stay findable.
Does ClaudeBot obey robots.txt?
Anthropic says its crawlers honour robots.txt and also support the non-standard Crawl-delay line. robots.txt is still a request: a server or CDN rule is the only thing that enforces a block.
How can I tell a real ClaudeBot request from a fake one?
Check the requesting address against the list Anthropic publishes at claude.com/crawling/bots.json. The user-agent string alone proves nothing.