Google expects your sitemap to list only live, canonical URLs. After a migration or a product cleanup, sitemaps quietly fill up with 404s and 301s, and Search Console starts complaining. Here's how to audit one in ~40 lines of Python.

1. Find the sitemaps

Start with robots.txt - most sites list their sitemaps there:

import re, requests def sitemaps_from_robots(origin): txt = requests.get(f"{origin}/robots.txt", timeout=20).text found = re.findall(r"(?im)^\s*sitemap:\s*(\S+)", txt) return found or [f"{origin}/sitemap.xml"]

2. Walk sitemap indexes and collect URLs 

import gzip, xml.etree.ElementTree as ET NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"} def urls_from_sitemap(url, depth=0): body = requests.get(url, timeout=30).content if url.endswith(".gz"): body = gzip.decompress(body) root = ET.fromstring(body) if root.tag.endswith("sitemapindex") and depth < 5: for loc in root.findall("s:sitemap/s:loc", NS): yield from urls_from_sitemap(loc.text.strip(), depth + 1) else: for u in root.findall("s:url", NS): yield u.findtext("s:loc", namespaces=NS).strip(), u.findtext("s:lastmod", namespaces=NS)

3. Check every URL politely 

def check(url): r = requests.head(url, allow_redirects=False, timeout=20) if r.status_code in (405, 501): # some servers don't support HEAD  r = requests.get(url, allow_redirects=False, timeout=20, stream=True) return r.status_code, r.headers.get("location")

Keep concurrency low (2-3 requests at a time per site), respect robots.txt disallow rules, and write rows with status >= 400 or a location header to a CSV. Those are the entries to fix or remove.

Edge cases you'll hit

  • Sitemap indexes nested several levels deep, .xml.gz files, and plain-text sitemaps.
  • robots.txt that returns an HTML error page with status 200.
  • Servers that answer HEAD with 404/405 but GET with 200.
  • Huge sitemaps (50k URLs/50 MB per file) - stream or cap them.

If you don't want to maintain it

Disclosure: I built a hosted version of exactly this, Sitemap Scraper on Apify, which handles the edge cases above, runs many sites at once and exports to CSV/Sheets ($0.20 per 1,000 URLs, $0.50 per 1,000 status checks). The script above works fine for one site too.