How to find a news site's feed or section pages from robots.txt and sitemaps in Python

A feed reader's first job is finding the feed, and for a lot of large news sites there isn't one to find. The fallback most people reach for is scraping the front page, which is fragile and impolite. There is a better order of things to try, and most of it is announced by the site itself: in a <link> tag, in robots.txt, and in sitemaps. The osFoundry desktop app does this in Rust (news_feed.rs, functions news_discover_feeds, news_discover_sections and news_fetch_section). Below is the same procedure in standard-library Python, run against eight real news sites, with the requests each one cost and where each one ended up.

Ground rules before the first request

A discovery script hits sites it has never seen, so the politeness has to be built into the one function that makes requests, not left to the caller.

UA = "Mozilla/5.0 (compatible; osFoundryNews/1.0; +https://osfoundry.io)" PAUSE = 1.0 # seconds between requests to the same site MAX_REQUESTS = 25 # per site, everything included  class Site: def __init__(self, host): self.root = f"https://{host}/" self.log, self.robots, self._last = [], None, 0.0 def get(self, url, step): if len(self.log) >= MAX_REQUESTS: raise Budget(url) if self.robots and not url.endswith("/robots.txt") and not self.robots.can_fetch(UA, url): self.log.append((step, url, "robots-disallow", 0, 0)) return None, None wait = self._last + PAUSE - time.monotonic() if wait > 0: time.sleep(wait) t0 = time.monotonic() req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept-Encoding": "gzip"}) status, body = None, None try: with urllib.request.urlopen(req, timeout=15) as r: raw = r.read(8 * 1024 * 1024) if r.headers.get("Content-Encoding") == "gzip" or url.endswith(".gz"): raw = gzip.decompress(raw) status, body = r.status, raw.decode("utf-8", "replace") except urllib.error.HTTPError as e: status = e.code except Exception as e: status = type(e).__name__ self._last = time.monotonic() self.log.append((step, url, status, len(body or ""), round((self._last - t0) * 1000))) return status, body 

One request at a time, at least a second apart, a hard cap per site, a User-Agent that says who is asking (the same one the app sends), and robots.txt checked through urllib.robotparser before every other URL. The log is what the numbers below come from.

robots.txt is fetched first, before the front page. That costs one request on sites that turn out to have a feed, but it means the rules apply to everything after it, and it carries the sitemap list that the later steps need anyway. Two more stop conditions sit on top: if robots.txt disallows the site root for this User-Agent, or the front page answers 401, 403 or 429, the script stops for that site.

Step 1: the feed the site announces

Sites with RSS almost always say so in the page head. One request for the front page, one HTML parse:

class Links(HTMLParser): def __init__(self): super().__init__() self.alternates, self.anchors = [], [] def handle_starttag(self, tag, attrs): a = dict(attrs) if tag == "link" and "alternate" in (a.get("rel") or "").lower().split(): t = (a.get("type") or "").lower() if ("rss" in t or "atom" in t) and a.get("href"): self.alternates.append(a["href"]) elif tag == "a" and a.get("href"): self.anchors.append(a["href"]) 

On The Verge, TechCrunch and ITmedia, this ended it: two requests each (robots.txt and the front page), done in under 4 seconds including the pause. ITmedia's feed lives on another host (rss.itmedia.co.jp), which is one reason to trust the announced link over guessing paths.

Step 2: the nine usual addresses

When nothing is announced, the app tries nine common feed paths and stops at the first one that parses as RSS or Atom: /feed, /feed/, /rss, /rss.xml, /feed.xml, /atom.xml, /index.xml, /feeds/posts/default, /rss/index.xml. These catch CMS defaults (WordPress, Blogger, Hugo, Ghost) on sites whose theme forgot the <link> tag.

On the three Japanese national papers in the test, Nikkei, Asahi and Yomiuri, all 27 probes failed: 26 answered 404 and one (asahi.com/rss) answered 403. That was 9 of the 13 or 14 requests each site cost. If you only care about large publishers, reading the sitemaps before these probes would cut each of those runs to 4 or 5 requests. For a general tool that also has to handle small blogs, the probes earn their place.

Step 3: what robots.txt points at

robots.txt lists sitemaps with Sitemap: lines. A sitemap is either a <sitemapindex> pointing at more sitemaps or a <urlset> of pages. The script reads up to four of the listed sitemaps, goes one level into an index (up to five children), and stops at the first one it can use.

What makes a sitemap usable is the part that needs care. The obvious test is the Google News namespace, xmlns:news=, which the format reserves for recently published articles. Nikkei shows why that test is not enough. Its sections_sitemap.xml declares the namespace, and 184 of its 187 URLs are section pages like /business/startups/. So the script judges a sitemap by the URLs in it:

def usable(info): """Judge a sitemap by the URLs in it, not by the namespace it declares.""" if info.get("section_share", 0) >= 0.7: return "sections" if info.get("article_share", 0) >= 0.7 and info["kind"] == "news_sitemap": return "news sitemap" if info.get("article_share", 0) >= 0.7 and info.get("dated"): return "article sitemap" return None 

section_share and article_share are the fractions of <loc> paths that pass the two shape tests below. The 70 % threshold is the one the app uses for section sitemaps.

Here is where each site ended up:

Measured from a Mac in November 2026, one site after another. A good part of the seconds is the one-second pause; the slowest single response was 6.9 s, the 403 from asahi.com/rss.

Yomiuri and Asahi need nothing more: a news sitemap is a feed without summaries, with a URL, a title and a publication time per article. Asahi's index splits it by desk (sitemap_national.xml, sitemap_business.xml and so on), so following it means picking desks, much like picking sections.

Step 4: section pages, when that's all there is

Nikkei's sitemap is a directory of the site rather than a list of articles. The shape test for a section is the inverse of the article test, plus a depth limit and a short blocklist:

def looks_like_article(path): segs = [s for s in path.split("/") if s] if not segs: return False last = re.sub(r"(\.html?)+$", "", segs[-1]) if len(last) < 6: return False if len(segs) < 2: return last.count("-") >= 3 if any(re.fullmatch(r"\d{4}", a) and re.fullmatch(r"\d{2}", b) for a, b in zip(segs, segs[1:])): return True if len(last) >= 8 and re.search(r"\d", last): return True return last.count("-") >= 3 def looks_like_section(path): if looks_like_article(path): return False segs = [s for s in path.split("/") if s] if not segs or len(segs) > 3: return False if "." in segs[-1] and not segs[-1].endswith(".html"): return False return not any(s.lower() in BLOCKED for s in segs) # search, login, about, privacy, ... 

An article is a path with a date pair in it (/2026/11/06/), an ID-like last segment of 8 or more characters containing a digit, or a slug with at least three hyphens. The ID rule is not applied at depth 1, because there it would swallow tag pages like /coronavirus-2026. Three hyphens rather than two, because /news/tech-and-science/ is a section on plenty of sites.

Following a section is then one request per check: fetch the section page, keep same-host links with an article shape, and drop strays.

shaped = [u for u in unique if looks_like_article(urlparse(u).path)] first = lambda u: next((s for s in urlparse(u).path.split("/") if s), None) counts = {} for u in shaped: counts[first(u)] = counts.get(first(u), 0) + 1 if counts: seg, n = max(counts.items(), key=lambda kv: kv[1]) if n * 10 >= len(shaped) * 6: shaped = [u for u in shaped if first(u) == seg] 

Nikkei's Startups page had 271 distinct same-host links. The shape test and the dominant-segment rule (if 60 % of matches start with the same first segment, here /article/, the rest go) left 20, all of the form /article/DGXZQOUC026TI0S6A101C2000000/. The 251 that went were menus, tag and theme pages, /login, /search, market data pages and similar.

The app arrives at the same place. Its section picker for www.nikkei.com reads "Showing 60 of 185 sections": the 184 from the sitemap plus the front page, which it adds only after checking that the front page itself carries article links.

One more thing Nikkei's section sitemap gives away: every section has a <lastmod>, and 157 of the 187 had changed in the previous 24 hours. A follower checking many sections can read that one file first and skip the section pages that haven't moved.

When the answer is no

AP answered 403 to the front page, so the script stopped after two requests. Without that rule it would have gone on to make 13 more requests that also returned 403, the result of an earlier version of this script. Reuters' robots.txt lists named crawlers it allows and disallows everyone else, so the script stopped after one request and never fetched the front page.

Neither is a puzzle to work around. A site that answers 403 to an identified client, or says no in robots.txt, has answered the question. The useful behaviour for a feed tool is to say so clearly and stop, which costs the site one or two requests instead of a crawl.

The osFoundry desktop app runs this discovery locally when you add a source in its News screen; the news firehose page describes the rest of what that screen does.