When 'regenerate your sitemap' is wrong
An audit reports that some of the URLs in your sitemap are broken, and tells you to regenerate the file so it contains only live, indexable, canonical pages. When those URLs really are 404s, that is correct and it is the most common sitemap defect there is. When they are not, it is the one piece of audit advice that can destroy the evidence you needed.
The case where following it would have hidden a fault
An insurer's crawl found fourteen URLs answering with a server error — all of them returning one identical short body, which is its own tell. Three separate findings named those same fourteen URLs: one at critical severity saying the server was failing, one saying the pages were empty stand-ins, and the sitemap one saying to regenerate the file.
A reader who started with the sitemap finding would have removed fourteen entries, and with them the only list pointing at the fault. The critical finding would then have been describing pages nothing referenced any more, on a site whose sitemap now looked clean.
The distinction is simple once stated and invisible in a list of "errors":
- A 404 means the page is gone and your sitemap is stale. Remove the entry. That is what the advice is for.
- A server error means the page is supposed to exist and your server is broken. Removing the entry fixes nothing and deletes the record.
Both appear under the same heading in most reports. They call for opposite actions.
Three more things that are not a stale sitemap
- A rate limit. On a food retailer, the audit's own crawl provoked 429s and the report named three broken sitemap URLs; a re-run minutes later found none. A 429 is the one refusal a crawler can cause simply by existing — a rate limit is not a block →.
- Nothing coming back at all. A status of zero is the crawler's note that it stopped waiting, not an answer from your server. Reporting it as "broken" describes one working page two contradictory ways in one document — "could not be reached" is not a diagnosis →.
- A host that hung up once. This one is ours. Auditing docketseo.app, the host answered dozens of requests and reset the connection on a single one. Two findings named that URL: one correctly calling it rate limiting rather than an outage, and this one telling us to drop from the sitemap a page that answered 200 in a browser the same minute.
All four share a test that is worth memorising. A host that answered is not gone, so a refusal from it says nothing about the URL. If the rest of your site came back, the sitemap entry is almost certainly fine.
Why this finding deserves more care than most
Nearly all audit advice is additive: write a description, add a tag, link to a page. If the finding was wrong you have spent an hour and added something harmless.
This one is subtractive. It asks you to remove a URL from the single document that tells search engines which pages you have — which is often exactly the document somebody will use later to work out what went wrong.
A remedy that deletes a record should demand a higher standard of proof than one that adds a tag. That is the whole argument of this page, and it applies well beyond sitemaps: treat "remove", "delete" and "regenerate" differently from "add".
Reading the finding on your own report
- Look at the status beside each URL, not the heading. If the report does not show you a status per URL, that is worth knowing about your tool before you act.
- Check whether the same URL appears in another finding. When it does, the louder finding is usually the true one and this is the echo. On that insurer's report the same fourteen URLs were in three.
- Re-run before you remove anything. A list that shrinks on the second run was partly about the run.
- Open one in a browser. Thirty seconds, and it settles every case above.
When the advice is exactly right
Keep the severity in view — this is a real and common defect, and three shapes of it deserve the fix as written:
- Genuine 404s, usually pages deleted without the sitemap being rebuilt.
- URLs that redirect. A sitemap should list destinations, not stops on the way.
- Pages you have deliberately excluded from indexing. Listing a page you have told search engines to ignore is a contradiction, and the sitemap is the easier half to fix.
The fixes that make it worse
- Regenerating the sitemap while the server is failing. Your generator will faithfully omit every page that could not be rendered, and you will have shipped the outage into the file.
- Removing URLs by hand from a generated file. The next build puts them back, and you have learned nothing except that the finding returns.
- Cutting the sitemap down until the finding clears. Which brings us to the obvious.
How to clear this finding completely
Publish a sitemap containing one URL that works. The finding goes to zero and so does the usefulness of the file — a sitemap's job is to list your pages, and a tool counting broken entries cannot notice the ones you never declared.
Which is the honest limit of the check: it can only be wrong about URLs you told it about. Nothing in a sitemap finding says anything about the pages missing from your sitemap altogether.
Where this sits in an audit
The registered checks are index.sitemap, which reads the file and the URLs in it,
and index.broken, which owns the server-error and unreachable findings that this one
kept duplicating. The identifiers on the findings themselves differ from both — worth knowing when
you search for one by name.
For the outbound version of the same confusion between a refusal and a dead page, see how to fix broken outbound links (and which are not) →.
Common questions
My sitemap URLs are reported broken but they load fine. Why?
Usually because the crawler got a refusal, a rate limit or a timeout rather than an answer. If the rest of your site came back during the same crawl, the host is working and the entries are almost certainly fine.
Should I remove a sitemap URL that returns a server error?
No. A 404 means the page is gone and the entry is stale; a server error means the page is supposed to exist and the server is broken. Removing the entry hides the fault instead of fixing it.
Why did three findings name the same URLs?
Because several checks can see the same failed fetch from different angles. When that happens the loudest finding is usually the true one and the others are echoes — act on the one whose remedy addresses the cause.
Should a sitemap list redirecting URLs?
No. A sitemap should list destinations rather than stops on the way, so a URL that redirects belongs in the file only as the address it redirects to.
Does a clean sitemap finding mean my sitemap is good?
Only that the URLs you declared are reachable. Nothing in the check can see the pages missing from the file, which is the more common and more expensive sitemap problem.