Robots.txt tester
Paste the address of any page. The tester fetches that site's robots.txt and reads it the way the crawlers do, then shows, for this exact page, who may crawl it and which line in the file decides.
| Crawler | Who | What it is for | This page |
|---|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode | ||
Bingbot | Microsoft | Bing, and Copilot, which crawls as Bingbot | |
Applebot | Apple | Siri and Spotlight suggestions | |
DuckDuckBot | DuckDuckGo | DuckDuckGo search | |
OAI-SearchBot | OpenAI | ChatGPT Search results | |
ChatGPT-User | OpenAI | ChatGPT fetching a page a user asked about | |
Claude-SearchBot | Anthropic | Claude's search results | |
PerplexityBot | Perplexity | Perplexity's index and citations | |
GPTBot | OpenAI | OpenAI model training | |
ClaudeBot | Anthropic | Anthropic model training | |
Google-Extended | Gemini training and grounding, not Search | ||
CCBot | Common Crawl | the open web corpus many models train on |
That was one file, read for one page. Docket runs 99 checks across your whole site, ranked by what to fix first, each with where it is and the change to make. The free download shows your score and the top problem in full.
$349once · or 4 × $87.25
For Macs with Apple silicon or Intel, macOS 12 or later
30-day refund, no conditions.
Intel Mac? Get the Intel version
How to read the answer
Allowed or Blocked is what the file tells that crawler about this
page. The rule beside it is the line that decided, and "named in your file" means a group addresses
that crawler by name, so your * rules no longer apply to it. If no line matches, the page
is allowed.
The rules follow RFC 9309, the robots.txt standard, and match the parser inside Docket. A crawler
obeys every group that names it; only if none does, it obeys the * groups. The longest
matching rule wins, and Allow wins a tie. * matches any run of characters and
a $ at the end anchors it. If the server answers robots.txt with a server error, crawlers
stay out of the whole site; if it answers "not found", everything is allowed.
What robots.txt does not do
- It does not remove a page from search. A blocked page can still be indexed from
links pointing at it, without its content. To keep a page out, allow crawling and add a
noindextag. - It is a request, not a lock. Well-behaved crawlers obey it. A firewall or CDN rule is what actually refuses a request, and this tester cannot see one.
- It is per host.
www.and the bare domain, and every subdomain, each have their own file. The tester reads the file for the host you type.
Checking every page
Docket reads robots.txt for every page it crawls, requests pages as each AI crawler to find server
and CDN refusals, and checks the noindex tags and headers this tester cannot see, then
ranks all of it with the rest of its 99 checks.
$349once · or 4 × $87.25
For Macs with Apple silicon or Intel, macOS 12 or later
On your phone?
Founding price $174.50 once · or 4 × $43.63, first 25 buyers, then $349. Buy applies the code FOUNDINGDOCKET for you; if the checkout shows $349, enter it under ‘Have a discount code?’.
Screaming Frog is $279 a year. Ahrefs plans with Site Audit run $29–$449 a month. Docket is $349 once. Their prices read 8 October 2026.
30 days, no conditions, no questions asked. Refund policy
Related
- AI crawler checker: can ChatGPT read your site?
- Which AI crawlers to block and which to keep
- Does noindex stop AI crawlers?
Common questions
How do I test my robots.txt?
Paste the address of a page into the tester above. It fetches the site's robots.txt, applies the rules the way crawlers do, and shows for each crawler whether that page is allowed and which line decided.
Does Disallow in robots.txt remove a page from Google?
No. It stops Googlebot fetching the page, but the page can still be indexed from links to it, without its content. To keep a page out of search, allow crawling and add a noindex tag.
Why is a page blocked when my * rules allow it?
Because a group names that crawler. A crawler that finds a group addressed to it by name obeys only that group and ignores the * rules entirely.