Does noindex stop AI crawlers?

By Matt Kerr·Published 15 September 2026

What the tag is actually addressed to

A noindex travels in one of two places: a robots meta element in the HTML, or an X-Robots-Tag response header. Both of them arrive inside a response. That single structural fact settles the question. The request was made, the server answered it, the bytes were transferred, and only then is there a tag for anything to read. Whatever the reader does next, the copy has already been taken.

For a search engine that honours the directive, what happens next is that the page is withheld from the results listing. That is a real and useful effect, and it is the effect the tag was designed for. It is also the wrong tool for the worry that brings most people to this question, which is not "will this page be listed" but "is this text being ingested".

The interaction runs the other way as well, and it is the reason the two controls should never be stacked casually. A URL disallowed in robots.txt is not fetched by a compliant crawler, so the noindex on it is never seen. Blocking access does not deliver a suppression instruction; it prevents one from being delivered. The directive side of this is AI crawler access, and the indexing side sits with canonical tags, which are a hint about duplicates rather than a removal instruction.

Three controls, three different jobs

Control	Stops indexing	Stops fetching	Who has to co-operate
noindex (meta robots or X-Robots-Tag)	Yes, in search engines that honour it	No. The page is requested and served in full first	The indexer, after it already holds your page
Disallow in robots.txt, per user-agent token	Not directly. A disallowed URL can still be listed from other signals	Yes, for crawlers that comply	The crawler, voluntarily, before it requests anything
Your server or CDN	Only as a side effect — there is nothing to index	Yes, for everybody. This is the enforcement layer	Nobody. The refusal is issued, not requested

Read down the "stops fetching" column and the answer to the page's question falls out. The only row that governs retrieval by consent is robots.txt; the only row that governs it absolutely is the server. The tag is in neither position.

Which crawler you are refusing matters more than whether you refuse

Docket's reports sort AI crawlers by the job they do rather than lumping them into one category, because the cost of blocking is wildly different across them. On the OpenAI side, GPTBot is the training crawler, OAI-SearchBot builds the search index that decides whether you can be cited, and ChatGPT-User fetches a page live when somebody asks about it. Refusing the first is an editorial position plenty of publishers hold on purpose. Refusing the second removes you from answers, which is rarely what anyone sat down intending.

Across the 6,236 hosts in the survey with a parseable robots.txt, 1,371 block at least one training crawler and 646 block at least one citation crawler. Two checks in the audit cover the two layers separately — ai.crawler_access reads what the directives say, and ai.edge_access asks the server what it actually does. Neither of them is a check on your meta tags, because a meta tag is not where this is decided.

The measurement: the edge answers first, and it answers alone

The survey read robots.txt for the Tranco top 10,000 and, separately, put a request to each server as a bot that said who it was. The gap between those two readings is the finding. 643 hosts — 6.43% — returned 401, 403, 406, 429 or 503 to that request. None of those refusals was produced by a directive or a tag; the connection was closed before either could matter.

The same pattern shows up one layer further in. Among hosts whose robots.txt explicitly permitted a fetch of /llms.txt, 4.73% were refused the file by the server anyway. A site can publish a permissive robots.txt, leave every page indexable, and still be unreachable to the crawlers it meant to welcome. If that is the shape of what you are seeing, the diagnosis is on when your CDN blocks AI crawlers your robots.txt allows, and the platform defaults behind it are on Cloudflare and GPTBot.

One switch, two losses

The sharper result in the survey is about what people intended. Of the 425 sites refusing OAI-SearchBot, 414 refuse GPTBot as well — 97.4%. Only 11 sites in the whole sample gave up search while still permitting training.

An overlap that tight is not the residue of many separate judgements. It is the signature of a single blanket switch: somebody decided to refuse training, the control they reached for said "AI bots", and citation went out with it. Nearly nobody chose that trade, and the ones who did are a rounding error. Separating the decision per crawler is the whole of the fix, and it costs nothing but attention.

The limit, stated plainly

robots.txt has no teeth. It is a published request that well-behaved crawlers choose to honour, and a crawler that ignores it faces nothing but its own operator's policy. Anything that must be refused rather than asked has to be refused at the server or the CDN — which is exactly why the edge figure above is the one worth acting on, and why a robots.txt audit on its own cannot tell you what your site does.

The honest counterpart is that the edge is blunt. A rule that turns away an unrecognised user-agent turns away the crawlers you wanted along with the ones you did not, and it leaves no trace in any file a reader of your site can inspect.

What to do instead

• Stop treating noindex as an access control. Use it for what it does: keeping a page out of search results. It is not a consent mechanism and was never offered as one.
• Decide per crawler, in robots.txt. Refusing a training crawler while admitting a search crawler is a coherent, statable position. A single category toggle cannot express it — and check the names you write, because more than half the files that write an AI rule at all address a crawler that no longer answers to it.
• Test the server, not the file. Request the same URL twice from one machine — once as an ordinary browser, once as the crawler — and compare. A browser 200 against a crawler 403 is the finding, and it is invisible to anything that only reads robots.txt.
• Check that a crawler you admit can read what it gets, which is a separate failure again — see JavaScript rendering and AI search visibility.
• Look at your own site the way an assistant does, rather than the way your CMS does: audit your site from an AI assistant.

Download Docket

Common questions

Does noindex stop AI crawlers?

No. A noindex directive is carried inside the response — either as a robots meta element or an X-Robots-Tag header — so the crawler has already requested and received the page by the time it can read the tag. noindex governs whether a search engine shows the page in results, not whether anything may fetch it.

What does stop an AI crawler, then?

Two layers. A Disallow rule in robots.txt against that crawler's user-agent token stops crawlers that choose to comply, before they request anything. Your server or CDN stops everything, because the refusal is issued rather than requested. robots.txt is a published request; the server is the only enforcement.

If I block a crawler in robots.txt, will it still see my noindex?

No, and that is the trap in stacking the two. A compliant crawler does not fetch a disallowed URL, so it never reads the noindex on it. Blocking access prevents a suppression instruction from being delivered rather than delivering one.

Does blocking GPTBot keep me out of AI answers?

Not by itself. GPTBot is the training crawler; OAI-SearchBot builds the search index that decides whether you can be cited, and ChatGPT-User fetches a page live when somebody asks about it. Refusing training while admitting search is a coherent position, though the survey suggests it is rarely the one taken: of 425 sites refusing OAI-SearchBot, 414 (97.4%) refuse GPTBot too, and only 11 gave up search while permitting training.

My robots.txt allows AI crawlers. Is that the whole answer?

No. In Docket's August 2026 survey of the Tranco top 10,000, 643 hosts (6.43%) refused a self-identifying bot outright — 401, 403, 406, 429 or 503 — before any robots rule could apply, and 4.73% of hosts whose robots.txt explicitly permitted a fetch of /llms.txt were then denied it by the server. Test the server, not the file.