AI search visibility, explained

AI search visibility is whether a language model can reach your website, read it, and quote it when someone asks a question your business could answer. It is not the same as ranking in Google, it is measured differently, and the first requirement is the one most sites get wrong by accident: letting the right crawlers in.

The three gates

A model has to clear all three before it can name you. They fail in order, so there is no point working on the third while the first is broken.

1. Access — can the crawler fetch the page?

Every AI search product runs its own crawler, and they are not interchangeable:

The distinction between a training crawler and a search crawler is the single most consequential thing on this page. Blocking training is a defensible decision about whether your writing becomes model weights. Blocking search removes you from the answer.

We measured this twice. Of 98 well-known sites with a robots.txt, 30% block at least one AI search crawler (the dataset). Then we read the robots.txt of the Tranco top 10,000, where the picture is better: of the 1,381 sites blocking any AI crawler, 53.2% blocked training and left search alone. Large sites mostly separate the two. What almost nobody catches is the third case — 51.9% of sites writing AI rules at all name a user-agent token that no crawler uses, so the rule they wrote does nothing.

2. Rendering — is there anything in the HTML?

Google renders JavaScript, eventually. Most AI crawlers do not run it at all. A React or Vue page whose content appears only after hydration is, to GPTBot and PerplexityBot, an empty document.

This is the failure that is hardest to notice, because the page looks perfect in a browser and perfect to Google. The test is simple: fetch your page with JavaScript disabled and see whether the words are there.

3. Extractability and entity clarity — can it quote you, and does it know who you are?

Once a model can read the page, two things decide whether it uses it. First, whether there is a passage it can lift: a question-shaped heading followed by a direct answer in the first two sentences is quotable, and eight paragraphs of preamble are not. Second, whether it can resolve your site to a real organisation — which is what sameAs in your Organization schema does, linking you to the LinkedIn, Wikidata or Google Business Profile the model already has an entry for.

Where the field currently stands

Blocking is concentrated almost entirely in publishing. News sites in our sample blocked AI search crawlers at 94.7%. SaaS companies were at 5.0%. Ecommerce and local businesses were in single digits.

For a business competing for customers rather than readers, that means visibility here is not a competitive advantage you can win — it is table stakes you can lose by accident, usually via a copied robots.txt block.

What about llms.txt?

llms.txt is a proposed convention for pointing models at your best content. Independent studies through 2026 have not found a measurable effect on citations, and Google has said it ignores the file. It costs nothing to add and there is no evidence it helps. Access, server-side rendering and entity clarity are where the measurable gains are.

The question after this one

Access, rendering and entity clarity decide whether a model can cite you. Whether it needs to is a different question, and it is the one with the uncomfortable answer: which of your pages an answer replaces outright.

How to check your own site

1. Open yoursite.com/robots.txt and look for each crawler above by name. If you only see User-agent: *, whatever it says applies to all of them.
2. Load a key page with JavaScript disabled. If the content vanishes, AI crawlers see nothing.
3. Check that your homepage has Organization schema with a sameAs array.

Or run Docket, which does all three plus 90 other checks and tells you which crawler you are blocking and what it costs you. It runs on your Mac and sends nothing anywhere.

Download Docket

Common questions

What is AI search visibility?

Whether an AI assistant can reach your website, read its content, and cite it when answering a user's question. It depends on crawler access in robots.txt, whether pages render server-side, and whether the site is identifiable as a real entity.

Is blocking GPTBot the same as blocking ChatGPT?

No. GPTBot collects training data. OAI-SearchBot builds the index ChatGPT searches when answering a question. Blocking GPTBot keeps you out of training; blocking OAI-SearchBot keeps you out of the answers.

Does llms.txt improve AI visibility?

There is no measured evidence that it does. Independent studies through 2026 found no effect on citations and Google has stated it ignores the file. Crawler access, server-side rendering and structured data are where the measurable gains are.