How many sites actually block Googlebot in robots.txt?

By Matt Kerr·Published 15 September 2026

Page one answers a different question than the one people ask

The query that led here — some version of "how many websites accidentally block Googlebot" — has a page one made entirely of repair manuals. Reading it on 15 September 2026 turned up 7 of them — tutorials on clearing "Blocked by robots.txt" out of Search Console — and nothing from Google at all. Every one is useful if you already know you have the problem.

Not one of them says how common the problem is. The genre takes the prevalence for granted: it is the premise that makes the tutorial worth writing, and it is never measured. That gap is what this page fills, and the answer turns out to cut against the premise.

What we measured, and what a share here is a share of

The survey asked the Tranco top 10,000 hosts for /robots.txt in August 2026 and recorded, per host, whether the file expressed a denial for each of a list of named crawler tokens. Most of that list is AI crawlers; Googlebot and GoogleOther are in it too, which is why this question can be answered at all.

The population is the first finding, and it belongs before the headline rather than in a footnote. Only about 62.4% of the hosts we asked produced a robots.txt this survey could read and parse. The rest split three ways:

Outcome	Hosts	Share of hosts attempted
Readable, parseable robots.txt — the denominator for every percentage below	6,236	62.4%
Unreachable: no answer, or an answer we could not use	2,332	23.3%
Reachable, but serving no robots.txt at all	1,316	13.2%
Excluded as oversize before the readable set was formed	116	1.2%
Attempted	10,000	—

Those 2,332 unreachable hosts are not evidence of anything about robots.txt. They are hosts that did not answer us, and a survey that quietly folded them into a denominator would be reporting its own reach as a property of the web. The 1,316 with no file at all are a real state and a permissive one — absent robots.txt means nothing is disallowed — but they express no directive, so they cannot be counted among files that do.

Which means the readable set is not a random sample of the top 10,000. Some of the hosts that refused our fetch refused it because they refuse automated fetches generally, and a host with that posture is not obviously less likely to disallow a crawler by name. So the rate below is a rate among files we could read, and the rate across the whole list could sit above it. We have no way to measure the difference, which is a reason to quote the figure with its denominator attached every time rather than a reason to quietly widen it.

The oversize bucket is excluded and stays excluded. Google documents a 500 KiB parse cap on robots.txt, and it is tempting to use a survey like this one to say how often real files approach it. We cannot. Those hosts were dropped before the readable set was formed, on a size threshold that is not in our repository, so nothing measured over the files we did read can speak for them. They are in the table because leaving them out would make the other rows add up wrongly, and for no other reason.

The headline, stated as narrowly as the data supports

Of the 6,236 readable files, 130 express a denial for Googlebot — 2.08%. Roughly one file in 48, in the most visible slice of the web there is.

Read that sentence literally, because it is written literally. It says the file expresses a denial. It does not say the site meant to be invisible in Google, and it does not say the site made a mistake. Those are claims about intent, and nothing in a robots.txt file carries intent. A publisher fencing off a staging host, a service that genuinely does not want indexing, and an engineer who shipped a Disallow line from a development environment all produce exactly the same bytes. We can count the bytes. We cannot read the room.

Against that, the same files are far more willing to refuse an AI crawler: 1,381 of them (22.1%) deny at least one of the 9 AI crawlers this survey tracks, and 646 (10.4%) deny at least one of the crawlers that decide whether a site can be cited in an AI answer. Blocking an AI crawler is roughly 10.6 times as common as expressing a denial for Googlebot.

One honest deflation of that comparison. The AI figure counts a host that denies any one of 9 tokens; the Googlebot figure counts one token. A count of "any of several" will beat a count of "this one" even if no site cared more about the category — some of the gap is arithmetic rather than sentiment. The direction survives the caveat, and the size of it should be read loosely. The full breakdown by token is on the AI directives survey, and the smaller hand-checked sample behind it is the Docket Index.

GoogleOther is the sharper half of this

GoogleOther is Google's general-purpose fetcher: the crawler used for product and research work other than building the Search index. It shares Googlebot's infrastructure and obeys its own token in robots.txt. Denying it does not remove a site from Search.

More files deny it than deny Googlebot — 305 against 130, about 2.3 times as many. We cannot show why, and will not guess — but the two tokens are genuinely different decisions, and the cost of getting them confused is asymmetric:

Token	What it fetches for	What denying it costs
Googlebot	The Search index	Pages stop being fetched for Search. This is the expensive one.
GoogleOther	Google product and research work other than the Search index	Those non-Search uses stop. Search is unaffected.

Sorting the readable files by which of the two tokens they deny leaves three shapes, and the way they are sized is the most useful thing on this page:

Shape	Files	What the file expresses
Both tokens denied	117	The commonest shape, and most files denying Googlebot are in it — the two often move together.
GoogleOther denied, Googlebot allowed	188	Non-Search fetching refused while Search fetching is permitted. The two tokens pulled apart — and the commoner way of pulling them apart, by a factor of about 14.5.
Googlebot denied, GoogleOther allowed	13	The rarest of the three shapes by a wide margin.

The asymmetry is what carries the argument. If "Google" were being treated as one undifferentiated switch, almost everything would sit in the top row and both of the other two would be marginal. One of them is not marginal at all. Files pull the two tokens apart routinely, and they pull them apart overwhelmingly in the direction that leaves Search fetching intact.

What we are not going to do is tell you the 188 files in the middle row made a considered choice, or that the 13 in the bottom row made a mistake. We cannot see either. A file denying Googlebot while allowing GoogleOther is simply the rarest of the three shapes, and some of those files may mean exactly what they say. The usable conclusion is narrower and holds regardless: these are two tokens, they are moved independently across this population, and a site treating them as one switch is expressing something it may not mean. If your file carries one of these lines and not the other, that is worth a deliberate look rather than a copy-paste.

What we tried to tell you about the blockers, and could not

The obvious follow-up is: what kind of site does this? We tried. A hostname-pattern pass over the blockers placed a handful of them as content delivery or advertising infrastructure — hosts where a blanket denial is unremarkable — and left the large majority unclassified. A handful out of a population this size supports no characterisation at all.

So the honest answer is that we do not know what kind of site typically expresses a denial for Googlebot, and we are not going to gesture at one. If somebody tells you it is mostly small business sites, or mostly WordPress, or mostly staging environments that leaked, ask what they counted.

What "Blocked by robots.txt" in Search Console actually means

The report the tutorials are written about says something narrow: Google wanted to fetch a URL, consulted your robots.txt, and your file said no. It is a statement about fetching. It is not a statement about whether the URL is in the index, and that trips people up constantly.

• robots.txt governs fetching, not indexing. A disallowed URL can still be listed in search results from other signals — links to it, for instance — typically with no description, because the page was never read. If your goal is that a page not appear, robots.txt is the wrong instrument, and the tag side of that argument is whether noindex stops a crawler.
• robots.txt binds only crawlers that read it and comply. It is a published request, honoured by the crawlers that choose to honour it. A denial is not a lock.
• Allowing a crawler in the file does not mean your server will serve it. A CDN rule, a bot-protection product or a rate limiter can refuse a request that robots.txt permits, and nothing in the file will show it. If that is your situation, the diagnosis is on fixing AI crawler access, and the evidence lives in your own logs — log file analysis is how you find out who actually fetched what.

How to check your own file

This takes about a minute and does not need a tool.

1. Read the file yourself at /robots.txt on the exact host and scheme you care about. A robots.txt applies to the host that served it, so the answer for www is not automatically the answer for the bare domain, and staging hosts have their own file.
2. Find every group heading, not just the first. Look for User-agent: Googlebot, User-agent: GoogleOther and User-agent: * separately, and read the Disallow lines under each. A crawler obeys the most specific group that names it and ignores the rest, so a permissive * group does not soften a restrictive Googlebot group.
3. Check the two Google tokens as two decisions. They are not interchangeable and a rule under one says nothing about the other.
4. Confirm it in Search Console rather than trusting your reading of the file. Google's own tooling reports what Google's parser concluded, which is the only interpretation that matters for Search.
5. Check that the server agrees with the file. A permissive robots.txt in front of an edge that returns a refusal is the failure mode nothing in the file can reveal.

Docket reads robots.txt as part of an audit, reports which named crawlers your file denies, and separately asks your server whether it will actually serve a request — the two halves that this page keeps insisting are different questions. The full list of what it inspects is on what Docket checks, and the sitemap side of robots.txt, which is a non-group record with rules of its own, is covered in AI crawlers and sitemaps.

What to take from this

If you arrived worried, the base rate is reassuring: 2.08% of readable files in the top 10,000 express a denial for Googlebot, so the ambient assumption that this is everywhere is not supported by the most visible slice of the web. That is not permission to skip the check on your own site — a base rate of one in 48 is exactly the sort of number that is comforting in aggregate and catastrophic in your particular case.

And if you find such a line, resist the urge to classify it as an accident before you have asked whoever wrote it. We could not make that call from the outside on 130 sites, and neither can a tutorial that never counted them.

Download Docket

Common questions

How many websites block Googlebot in robots.txt?

In Docket's August 2026 survey of the Tranco top 10,000, 130 of the 6,236 readable robots.txt files expressed a denial for Googlebot — 2.08%. That share is of the readable files only: 2,332 hosts were unreachable, 1,316 served no robots.txt at all, and 116 were excluded as oversize before the readable set was formed.

How many of those blocks are accidents?

We cannot tell you, and neither can anyone else working from robots.txt alone. The file records what it expresses, not what anyone intended: a deliberate policy and a line shipped from a development environment produce identical bytes. Any figure presented as a rate of accidental Googlebot blocking is a guess wearing a percentage sign.

Is blocking GoogleOther the same as blocking Googlebot?

No. Googlebot fetches for the Search index; GoogleOther is Google's general-purpose fetcher for product and research work other than building that index, and it obeys its own token. Denying GoogleOther does not remove a site from Search. More files deny it than deny Googlebot — 305 against 130, about 2.3 times as many. The files also pull the two apart in one direction far more than the other: 188 deny GoogleOther while allowing Googlebot, against 13 the other way round, with 117 denying both. Which is why treating 'Google' as one switch is worth a second look.

Does a Disallow line keep a page out of Google's index?

Not reliably. robots.txt governs fetching, not indexing: a disallowed URL can still be listed from other signals, usually without a description because the page was never read. robots.txt also binds only the crawlers that read it and comply, and a file that allows a crawler says nothing about whether your server will actually serve it.

Do more sites block AI crawlers than block Googlebot?

Yes, by a wide margin in this survey: 1,381 readable files (22.1%) deny at least one of the 9 AI crawlers tracked, and 646 (10.4%) deny at least one crawler that feeds AI answers, against 130 (2.08%) for Googlebot. Read the gap loosely: the AI figure counts a denial of any one of several tokens while the Googlebot figure counts one, so some of the difference is arithmetic rather than sentiment.