Extract every URL from a sitemap.xml in Python (indexes, .gz, robots.txt) and spot competitors' new pages

A site's sitemap is the cheapest full list of its pages you will ever get: the site publishes it for search engines, it usually carries a lastmod date per URL, and reading it takes seconds instead of crawling HTML for hours. That makes it the right starting point for SEO audits, migrations, content inventories, feeding a crawler or RAG pipeline, and keeping an eye on what competitors publish.

The catch is that "the sitemap" is rarely one file. Here's what you actually run into, and a small Python script that handles it.

What real sitemaps look like

• Discovery: the reliable place to look is robots.txt, which lists sitemaps on Sitemap: lines (often several). If it has none, try the usual paths: /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml (WordPress core), /sitemap.xml.gz.
• Sitemap indexes: big sites publish a <sitemapindex> that points to more sitemaps, and indexes can point to other indexes. Yoast, Rank Math, Shopify and most frameworks split by post type or by 50,000 URLs (the per-file limit in the protocol).
• Gzip: .xml.gz files are common on news and e-commerce sites. Don't trust the Content-Type header; check for the gzip magic bytes 1f 8b instead.
• Namespaces: every tag is namespaced ({http://www.sitemaps.org/schemas/sitemap/0.9}loc), and image, video, news and xhtml:link hreflang extensions add more. Strip the namespace and match on the local name.
• No sitemap at all: some well-known sites don't publish one. In my test python.org returned nothing from robots.txt or the common paths, so you need a crawler for those.
• Loops and duplicates: some indexes list themselves, and the same URL can appear in several sitemaps. Track what you've visited and dedupe.

The script

Only dependency: pip install requests. Standard-library xml.etree does the parsing.

import gzip, json, pathlib, re, sys, time import xml.etree.ElementTree as ET from urllib.parse import urljoin import requests UA = {"User-Agent": "sitemap-tutorial/1.0"} COMMON = ["/sitemap.xml", "/sitemap_index.xml", "/wp-sitemap.xml", "/sitemap.xml.gz"] def find_sitemaps(site): site = site if site.startswith("http") else "https://" + site found = [] try: r = requests.get(urljoin(site, "/robots.txt"), headers=UA, timeout=20) if r.ok: found = [l.split(":", 1)[1].strip() for l in r.text.splitlines() if l.lower().startswith("sitemap:")] except requests.RequestException: pass if found: return found for path in COMMON: # fall back to the usual locations  u = urljoin(site, path) try: r = requests.get(u, headers=UA, timeout=20) if r.ok and b"<" in r.content[:200]: return [u] except requests.RequestException: pass return [] def fetch_xml(url): r = requests.get(url, headers=UA, timeout=30) r.raise_for_status() body = r.content if body[:2] == b"\x1f\x8b": # gzipped, whatever the Content-Type says  body = gzip.decompress(body) return ET.fromstring(body) def local(tag): # strip the XML namespace: {http://...}loc -> loc  return tag.rsplit("}", 1)[-1] def walk(url, seen=None): seen = seen if seen is not None else set() if url in seen: # some indexes list themselves  return seen.add(url) root = fetch_xml(url) if local(root.tag) == "sitemapindex": for sm in root: loc = next((c.text.strip() for c in sm if local(c.tag) == "loc" and c.text), None) if loc: yield from walk(loc, seen) else: for u in root: row = {local(c.tag): (c.text or "").strip() for c in u if local(c.tag) in ("loc", "lastmod")} if row.get("loc"): yield row def all_urls(site): urls = {} for sm in find_sitemaps(site): try: for row in walk(sm): urls[row["loc"]] = row.get("lastmod") except Exception as e: print(f" skipped {sm}: {e}", file=sys.stderr) return urls def new_since_last_run(site, urls, state=pathlib.Path("seen.json")): seen = json.loads(state.read_text()) if state.exists() else {} old = set(seen.get(site, [])) seen[site] = sorted(urls) state.write_text(json.dumps(seen)) return [u for u in urls if u not in old] if old else [] if __name__ == "__main__": for site in sys.argv[1:]: t = time.time() urls = all_urls(site) print(f"{site}: {len(urls)} URLs in {time.time() - t:.1f}s") 

Run it with one or more sites:

python3 sitemap_urls.py crawlee.dev docs.apify.com allbirds.com stripe.com 

What I got just now (October 20, 2026):

Gzip works the same way: walk("https://www.nytimes.com/sitemaps/new/news.xml.gz") returned 621 news URLs in 0.2 s, the newest with a lastmod only minutes old.

One thing to watch: robots.txt lives at the root, so asking for wordpress.org/news gave me 189,145 URLs for the whole of wordpress.org in 53 s. Filter on the path afterwards ([u for u in urls if "/news/" in u]) when you only care about one section.

Spotting new pages on a schedule

new_since_last_run() saves the URL set per site in seen.json and returns only the URLs it hasn't seen before. The first run records a baseline and returns nothing; after that, run it daily (cron, GitHub Actions) and you get a list of a competitor's new blog posts, product pages or docs:

urls = all_urls("competitor.com") for u in new_since_last_run("competitor.com", urls): print("NEW:", u, urls[u]) 

To check it, I removed five URLs from the saved state and re-ran: exactly those five came back as new.

Ideas for extending it:

• Filter by lastmod ("changed in the last 7 days") to find refreshed content, not just new content.
• Read xhtml:link alternates to audit hreflang coverage across languages.
• Feed the URL list into a bulk PageSpeed or broken-link check.
• Send the new URLs to Slack or a Google Sheet.

If you'd rather not run it yourself

Disclosure: I'm with Datawright Labs, and we packaged this (plus text, RSS and Atom sitemaps, hreflang, image and Google News fields, URL and date filters, and the new-URL monitoring) as a hosted tool on Apify: Sitemap URL Extractor. It costs $0.20 per 1,000 URLs, which Apify's free monthly credit covers for most small jobs. You can run it on a schedule and get results by API, webhook or CSV. The script above is the whole idea, though, so use whichever fits.