How to fix AI crawler access

To let AI assistants cite you while keeping your content out of model training, you need separate robots.txt rules for the search crawlers and the training crawlers. Most sites that try this end up blocking both, because the user-agent names do not make the difference obvious.

Which crawler does what

Each AI company runs at least two, and they serve different purposes:

Google-Extended is worth singling out: blocking it has no effect on your normal Google Search ranking, which is governed by Googlebot. People block it fearing otherwise.

Allow citation, refuse training

This is the configuration most businesses actually want. Paste it above any existing rules:

# AI search crawlers — these decide whether we can be cited.
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

# Training crawlers — opt out of model training.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

The mistake that causes this

A single blanket block:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Disallow: /

Consecutive User-agent lines form one group, so the Disallow applies to all four. This snippet circulated widely in 2024 as "block AI crawlers" and it removes you from ChatGPT and Perplexity results as well as from training.

In our measurement of 98 major sites, 68% of those blocking any AI crawler had also blocked the search crawlers — but reading the robots.txt of the Tranco top 10,000 later showed the opposite at scale: of 1,381 sites blocking any AI crawler there, 53.2% blocked training and left search alone. The data is here.

What allowing them does not do

Allowing a crawler is necessary for citation. It is not sufficient, and it is worth setting expectations before you go looking for results.

Two other things decide whether you actually get quoted. The first is rendering: most AI crawlers do not execute JavaScript, so a page whose content appears only after hydration is an empty document to them no matter what robots.txt says. The second is whether there is anything quotable — a heading phrased as the question someone asked, followed by a direct answer in the first two sentences, gets lifted; eight paragraphs of preamble do not.

There is also a timing reality. Search indexes refresh on their own schedule, so a robots.txt change made today does not produce citations tomorrow. Allow the crawlers, then judge it over weeks rather than days.

Check it worked

Rules resolve by longest match, not by order, so a later Disallow: / under User-agent: * does not override an earlier specific Allow — but a longer path pattern does. This is where hand-checking gets unreliable.

Docket parses robots.txt the way Google does and tells you, per crawler, whether it can reach your site and what blocking it actually costs. That check is one of 93 and runs in the first few seconds of any audit.

Download Docket

Common questions

How do I let ChatGPT read my website?

Allow OAI-SearchBot in robots.txt. That is the crawler that builds ChatGPT Search's index. GPTBot is a separate crawler used for training and can be blocked without affecting whether ChatGPT can cite you.

Does blocking GPTBot stop ChatGPT citing my site?

No. GPTBot collects training data. Citation in ChatGPT Search depends on OAI-SearchBot, which is a separate user-agent and needs its own rule.