Can AI crawlers read your site?
Type your address. The checker reads your robots.txt the way the crawlers do and shows, for each of the 21 crawlers below, whether it may fetch your page. The ones that decide whether ChatGPT, Claude, Perplexity, Google's AI Overviews and Copilot can find and quote you come first, and the ones that only collect training data come second.
That was one file, read for one page. Docket runs 99 checks across your whole site, ranked by what to fix first, each with where it is and the change to make. The free download shows your score and the top problem in full.
$349once · or 4 × $87.25
For Macs with Apple silicon or Intel, macOS 12 or later
30-day refund, no conditions.
Intel Mac? Get the Intel version
Crawlers that decide whether AI can cite you
Block one of these and that assistant cannot read your pages when it answers a question, so it cannot quote or link you. For most businesses these should be open.
| Crawler | Who | What a block costs you | Your site |
|---|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode, which follow your Googlebot rules rather than any AI-specific token. | ||
Bingbot | Microsoft | Bing search. Microsoft's Copilot assistant crawls as Bingbot, so this is the rule it reads. | |
Google-CloudVertexBot | Blocks Vertex AI grounding on your content for Google Cloud customers who ask it to index your site. | ||
ChatGPT-User | OpenAI | Stops ChatGPT fetching your page when a user asks about you. | |
OAI-SearchBot | OpenAI | Blocks your pages from ChatGPT Search results entirely. | |
Claude-SearchBot | Anthropic | Blocks your pages from Claude's search results. | |
Claude-User | Anthropic | Stops Claude fetching your page on a user's behalf. | |
Perplexity-User | Perplexity | Stops Perplexity fetching your page on a user's behalf. | |
PerplexityBot | Perplexity | Removes you from Perplexity's index and citations. | |
Amazonbot | Amazon | Blocks Alexa and Amazon's answer surfaces. | |
Meta-ExternalFetcher | Meta | Stops Meta AI fetching your page on a user's behalf. | |
MistralAI-Index | Mistral | Removes you from Le Chat's search index and citations. |
Crawlers that collect training data
These gather text to train future models. Blocking them is a reasonable choice for many publishers, and it does not stop an assistant from finding and citing you through the crawlers above.
| Crawler | Who | What a block costs you | Your site |
|---|---|---|---|
Google-Extended | Blocks Gemini model training and grounding in Gemini Apps and Vertex AI. Does NOT affect AI Overviews, AI Mode or classic Google Search. Those follow your Googlebot rules. | ||
GoogleOther | Blocks Google's non-Search crawls. Does not affect Search. | ||
GPTBot | OpenAI | Blocks OpenAI from training on your content. Does not affect whether ChatGPT can cite you live. | |
ClaudeBot | Anthropic | Blocks Anthropic from training on your content. | |
Applebot-Extended | Apple | Blocks Apple Intelligence training. Applebot (search) is separate. | |
Bytespider | ByteDance | Blocks ByteDance/TikTok AI crawling. | |
CCBot | Common Crawl | Removes you from Common Crawl, an input to many open models. | |
meta-externalagent | Meta | Blocks Meta AI training and grounding. | |
MistralAI-Training | Mistral | Blocks Mistral from training on your content. |
What this checker cannot see
It reads one file, and robots.txt is a request, not a lock. Four things it cannot tell you:
- Whether your server or CDN lets the crawler in. A firewall, a bot-protection setting or a CDN rule can refuse a crawler that robots.txt welcomes. In Docket's August 2026 survey of the Tranco top 10,000, 643 hosts (6.43%) refused a self-identifying bot outright before any robots rule applied.
- Tags on the page itself. A
noindexornosnippetin the page or its headers limits what a search engine may show, and robots.txt says nothing about it. - Pages that need JavaScript to show their text. Most AI crawlers do not run it, so a page can be open to them and still look empty.
- What a crawler actually does. The file asks; well-behaved crawlers obey. It is not enforcement.
It checks the path you type, or your homepage if you type none. Rules for other sections of your site can differ.
How it reads the file
The rules follow RFC 9309, the robots.txt standard, and match the parser inside Docket, so the
two give the same answer for the same file. A crawler obeys every group that names it; only if none
does, it obeys the * groups. The longest matching rule wins, and Allow
wins a tie. If your server answers robots.txt with a server error, crawlers stay out entirely; if
it answers "not found", they may crawl everything.
The checker fetches only your robots.txt, names itself in the request, keeps a copy for a few minutes so repeated checks do not hit your server, and stores nothing about you or your site.
Checking every page, and your server
Docket checks all of this from your Mac, on every page it crawls: it requests pages as each AI crawler and reports when your server refuses one that robots.txt permits, reads the tags on every page, and flags text that only appears after JavaScript runs. It then ranks what to fix first, next to five optional outside checks and the rest of the audit.
$349once · or 4 × $87.25
For Macs with Apple silicon or Intel, macOS 12 or later
Founding price $174.50 once · or 4 × $43.63, first 25 buyers, then $349. Buy applies the code FOUNDINGDOCKET for you; if the checkout shows $349, enter it under ‘Have a discount code?’.
Screaming Frog is $279 a year. Ahrefs plans with Site Audit run $29–$449 a month. Docket is $349 once. Their prices read 8 October 2026.
30 days, no conditions, no questions asked. Refund policy
Related
- Which AI crawlers to block and which to keep
- When your CDN blocks AI crawlers robots.txt allows
- Does noindex stop AI crawlers?
- The Docket Index: who blocks AI search crawlers
Common questions
Is ChatGPT allowed to crawl my website?
Check OAI-SearchBot and ChatGPT-User in the table above. OAI-SearchBot builds the index ChatGPT Search cites from, and ChatGPT-User fetches a page when someone asks about it. GPTBot is OpenAI's training crawler; blocking it does not remove you from ChatGPT's answers.
Does blocking Google-Extended keep me out of AI Overviews?
No. Google-Extended governs Gemini training and grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode follow your Googlebot rules.
My robots.txt allows every AI crawler. Am I visible?
Not necessarily. A CDN, firewall or bot-protection setting can refuse a crawler that robots.txt allows, and this checker cannot see that. Docket requests pages as each crawler to find out.