How to fix AI crawler access

By Matt Kerr·Published 6 August 2026·Updated 5 November 2026

To let AI assistants cite you while keeping your content out of model training, you need separate robots.txt rules for the search crawlers and the training crawlers. Most sites that try this end up blocking both, because the user-agent names do not make the difference obvious.

Which crawler does what

Each AI company runs at least two, and they serve different purposes:

Google-Extended is worth singling out. Blocking it has no effect on Google Search, AI Overviews or AI Mode, which all follow your Googlebot rules. It controls whether Gemini trains on your pages and grounds answers in Gemini Apps, and nothing else. People block it fearing otherwise, and others leave it open thinking it is how they get into AI Overviews.

Allow citation, refuse training

This is the configuration most businesses actually want. Paste it above any existing rules:

# AI search crawlers: these decide whether we can be cited.
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training crawlers: opt out of model training.
# Google-Extended is Gemini training and Gemini Apps grounding. AI Overviews follow Googlebot.
User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

The mistake that causes this

A single blanket block:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Disallow: /

Consecutive User-agent lines form one group, so the Disallow applies to all four. This snippet circulated widely in 2024 as "block AI crawlers" and it removes you from ChatGPT and Perplexity results as well as from training.

In our measurement of 98 major sites, 68% of those blocking any AI crawler had also blocked the search crawlers, but reading the robots.txt of the Tranco top 10,000 later showed the opposite at scale: of 1,381 sites blocking any AI crawler there, 53.2% blocked training and left search alone. The data is here.

What allowing them does not do

Allowing a crawler is necessary for citation. It is not sufficient, and it is worth setting expectations before you go looking for results.

Two other things decide whether you actually get quoted. The first is rendering: most AI crawlers do not execute JavaScript, so a page whose content appears only after hydration is an empty document to them no matter what robots.txt says. The second is whether there is anything quotable. A heading phrased as the question someone asked, followed by a direct answer in the first two sentences, gets lifted; eight paragraphs of preamble do not.

There is also a timing reality. Search indexes refresh on their own schedule, so a robots.txt change made today does not produce citations tomorrow. Allow the crawlers, then judge it over weeks rather than days.

Check it worked

Rules resolve by longest match, not by order, so a later Disallow: / under User-agent: * does not override an earlier specific Allow, but a longer path pattern does. This is where hand-checking gets unreliable.

For one site and no download, the free AI crawler checker reads your robots.txt and shows, crawler by crawler, which of the ones above may fetch your page and which line decides.

Docket parses robots.txt the way Google does and tells you, per crawler, whether it can reach your site and what blocking it actually costs. That check is one of 99 and runs in the first few seconds of any audit.

Common questions

How do I let ChatGPT read my website?

Allow OAI-SearchBot in robots.txt. That is the crawler that builds ChatGPT Search's index. GPTBot is a separate crawler used for training and can be blocked without affecting whether ChatGPT can cite you.

Does blocking GPTBot stop ChatGPT citing my site?

No. GPTBot collects training data. Citation in ChatGPT Search depends on OAI-SearchBot, which is a separate user-agent and needs its own rule.