Docket / Learn / Crawlers behind Claude’s search

Which crawlers feed Claude’s web search?

By Matt Kerr·Published 6 November 2026

Claude’s web search reads pages through Anthropic’s own crawlers and through outside search providers. Anthropic’s are Claude-SearchBot and Claude-User. Anthropic’s subprocessor list also names three web search providers: Brave Search, Parallel Web Systems and turbopuffer. Parallel’s crawler is ShapBot. Brave’s crawler has no name of its own and follows your Googlebot rules.

Each one, in its operator’s words

Crawler	Operator	What the operator says	What robots.txt does
`Claude-SearchBot`	Anthropic	“navigates the web to improve search result quality for users”	Its own group, or `*`, decides.
`Claude-User`	Anthropic	fetches a page “when individuals ask questions to Claude”	Its own group, or `*`, decides.
`ShapBot`	Parallel Web Systems	“collects publicly available web content to build and maintain Parallel’s search index”	Follows a `ShapBot` group, or `*`. Parallel says changes take effect within about a day.
`Shap-User`	Parallel Web Systems	identifies itself “when accessing content on behalf of users”	Parallel says it is meant for visibility, not to be managed by automated controls, so a block may not stop it.
Brave Search’s crawler	Brave	“does not advertise a differentiated user agent”	Brave says a page Googlebot may not crawl is not crawled by Brave either.

Read on 6 November 2026 from Anthropic’s subprocessor list, Parallel’s bot page, effective the same day, and Brave’s crawler page. Anthropic’s own crawlers are from its help article.

turbopuffer is on the list too. It makes search database software, and Anthropic does not say what part it plays in Claude’s search. Anthropic does not publish how Claude weighs any of these providers either, so the safe position is to be open to all of them.

How to stay findable

If your robots.txt allows everything, there is nothing to change. If it closes the door to bots it does not know, which is common after a wave of AI blocking, name the search crawlers and open them:

User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ShapBot
Allow: /

Keep Googlebot open as well, because Brave follows those rules. Blocking ClaudeBot, which collects training data, does not affect any of this; what ClaudeBot does explains the difference. A firewall or CDN rule can still refuse a crawler your file welcomes, and that is the most common reason a site that meant to stay open is not.

Docket checks them

The free AI crawler checker now includes ShapBot and Shap-User beside Anthropic’s crawlers, and Docket’s app adds them in its next release.

Check your own site

Most sites have never checked what their file says to the crawlers behind Claude’s search. The free AI crawler checker reads your robots.txt the way the crawlers do and shows, crawler by crawler, which ones may fetch your pages and which rule decided it.

Check my site’s AI crawlers

$59once

For Macs with Apple silicon or Intel, macOS 12 or later, and 64-bit Windows

Buy Docket Try it free Try it free for Windows See a full sample report

On a Mac? Get the Mac version.

On your phone?

Black Friday price, all of November: Docket SEO $29.50 once. From 1 December it’s $59, or 4 × $14.75. Buy applies the code FOUNDINGDOCKET for you; if the checkout shows $59, enter it under ‘Have a discount code?’.

Screaming Frog is $279 a year. Ahrefs plans with Site Audit run $29–$449 a month. Docket is $59 once. Screaming Frog’s price read 4 November 2026, Ahrefs’ 28 October 2026.

30 days, no conditions, no questions asked. Refund policy

Common questions

Does Claude use Brave Search?

Anthropic's subprocessor list names Brave Search as a web search provider for all its products. Brave's crawler has no separate user agent and follows your Googlebot rules.

What is ShapBot?

Parallel Web Systems' crawler. Parallel says it collects public web content to build its search index, and Anthropic lists Parallel as a web search provider for Claude.

Should I block ShapBot?

Only if you want to stay out of Parallel's index. Parallel says restricting it may reduce your visibility through its search features, and Claude's web search uses Parallel.