Log file analysis: what Googlebot actually fetched
A crawl tells you what a search engine could reach on your site; a server log tells you what it actually fetched, how often, and what it wasted requests on. The two disagree constantly, and the single most common mistake in reading a log is trusting the user-agent — which is why Google publishes 1,641 IP prefixes across 3 lists so you can check.
The user-agent is a claim, not an identity
Every line in your access log carries whatever user-agent the client chose to send. "Googlebot" is a string, and scrapers send it constantly — partly to get past blocks, partly because pretending to be a search engine is the easiest way to look legitimate in somebody else's log file.
So a report that says "Googlebot fetched 40,000 pages last month" is really saying "40,000 requests claimed to be Googlebot". Those are different numbers and only one of them tells you anything about your crawl budget.
There are two honest ways to resolve it. Google publishes its crawler ranges as JSON — as of 2026-08-08, 315 prefixes for Googlebot itself (169 IPv4 and 146 IPv6), plus separate lists for special-case crawlers and user-triggered fetches, 1,641 in total. Or you do the check Google documents: a reverse DNS lookup on the client IP, then a forward lookup on the hostname it returns, and the original address has to come back.
The second step is the one people skip, and skipping it defeats the whole exercise. A PTR
record is set by whoever controls the address block, so anyone can make their IP claim to be
crawl-something.googlebot.com. Only resolving that name back proves it.
Docket does both, and there is a test that fails if the forward lookup is ever removed.
Docket's verification was spot-checked against 6 addresses taken from Google's published Googlebot ranges — addresses that are Googlebot by Google's own definition — and accepted 6 of 6. That is a smoke test against real DNS rather than a rate; six addresses cannot be turned into a percentage and this page is not going to pretend otherwise.
What the log tells you that a crawl cannot
Run docket logs access.log --url https://example.com and it crawls the site as
well, then reports the two ways the sets disagree.
Pages Google fetched that the crawl never found. Orphans with no internal link pointing at them, URLs you retired that Google is still retrying, pages that exist only in the sitemap. Each is a page a crawler alone would never show you, because a crawler starts at the homepage and follows links.
Pages you link to that Google did not fetch. Careful here, and Docket prints the caveat next to the number: a log covers a period. Absence from one week of logs means Google did not fetch that page in that week. It does not mean Google cannot see it, and a tool that lets you read it that way is setting you up to go fixing a problem you do not have.
And the response codes Google actually received. Every redirect and every 404 in that list is a request spent on your site that returned nothing indexable — the clearest measure of wasted crawl budget there is, and it is measured rather than modelled.
Where the dedicated tool is better
Screaming Frog sells a Log File Analyser as a separate product, $139 per year with a free tier capped at 1,000 log events. It is a much deeper tool than what Docket does: a real interface for exploring the data, saved projects, imports that handle far more formats, and analysis over time rather than a single comparison. If log analysis is a regular part of your work, buy it — it is built for that and Docket is not.
What Docket gives you is the one comparison that answers "is Google spending its time on my important pages", included in the $149 one-time price rather than as a second annual subscription. That is the whole claim, and it is deliberately narrow.
What Docket deliberately does not do here
It reads Common and Combined format, and gzip. Anything else is counted as unparsed and reported as a number, because a parser that quietly skips a third of a file produces confident-looking statistics about the rest. It does not follow sessions, does not chart anything over time, and does not store your logs — it reads the file, prints, and exits.
It also does not do reverse DNS unless you ask. --verify is a lookup per
distinct address, so it is off by default and everything says "claiming to be Google" until
you turn it on.
Common questions
How do I know if it was really Googlebot in my logs?
Not from the user-agent, which is a header anybody can send. Either check the client IP against Google's published crawler ranges — 1,641 prefixes across 3 lists as of 2026-08-08 — or do a reverse DNS lookup on the IP followed by a forward lookup on the hostname it returns, confirming the original address comes back. The second step matters: a PTR record is set by whoever controls the address block, so a reverse lookup alone proves nothing.
What can a server log tell me that a crawl cannot?
Which pages a search engine actually fetched and how often, which is a different question from which pages it could reach. That surfaces orphan pages with no internal links, retired URLs still being retried, and the share of requests that returned a redirect or an error — crawl budget spent on nothing indexable.
If a page is missing from my logs, can Google not see it?
No, and treating it that way sends you fixing a problem you may not have. A log covers a period. A page absent from one week of logs was not fetched that week; it may be crawled rarely because it changes rarely. Look for it across a longer window before concluding anything.
Does Docket replace a dedicated log file analyser?
No. Screaming Frog's Log File Analyser is a separate product at $139 per year and is much deeper — a real interface, saved projects, more formats, analysis over time. Docket does one comparison, log against crawl, included in its one-time price. If log analysis is a regular part of your work, the dedicated tool is the right purchase.