Experiment: what share of the links in your GitHub READMEs still work? Measure every repo you own in 15 minutes

A README gets written once, usually the week a project is created, and then it sits there. The project evolves; the README's links don't. Badges point at retired CI services. "See the docs" links go to a docs site that moved. The demo link is a free hosting app that was deleted years ago.

I don't know what share of your README links are broken, and I'm not going to guess. Instead, here's an experiment that measures it for every public repo you own, in about fifteen minutes. You'll end up with a per-repo table, and probably a few surprises.

What you need

The GitHub CLI (gh), logged in to your account
Python 3.10+ with requests (pip install requests)
A cup of coffee while it runs

Step 1: list your repos and fetch their READMEs

The GitHub CLI can list repos and call the API. The README endpoint returns the file's content base64-encoded:

gh repo list YOUR_USERNAME --limit 200 --no-archived --source \ --json nameWithOwner -q '.[].nameWithOwner' > repos.txt wc -l repos.txt

--source skips forks, so you only measure READMEs you wrote. Remove it if you want forks included. Then fetch each README:

mkdir -p readmes while read -r repo; do safe=${repo//\//__} gh api "repos/$repo/readme" -q .content 2>/dev/null | base64 -d > "readmes/$safe.md" \ || echo "no README: $repo" done < repos.txt

Step 2: extract the links

READMEs mix Markdown links, bare URLs, image badges and HTML. A pragmatic regex catches the vast majority:

import re, pathlib, json URL = re.compile(r'https?://[^\s<>()"\'\]]+') def links(text): found = set() for m in URL.finditer(text): url = m.group(0).rstrip('.,;:!?*_`') found.add(url) return found data = {} for p in pathlib.Path("readmes").glob("*.md"): data[p.stem.replace("__", "/")] = sorted(links(p.read_text(errors="ignore"))) json.dump(data, open("links.json", "w"), indent=2) print(sum(len(v) for v in data.values()), "links in", len(data), "READMEs")

This deliberately ignores relative links (like docs/setup.md), which point inside the repo. Checking those is a separate exercise.

Step 3: check them, politely

We'll use HEAD first and fall back to GET, because some servers handle HEAD badly. We'll also limit concurrency per host so we don't hammer anyone.

import json, requests, concurrent.futures as cf from urllib.parse import urlsplit from collections import defaultdict import threading data = json.load(open("links.json")) session = requests.Session() session.headers["User-Agent"] = "readme-link-experiment/1.0" host_locks = defaultdict(threading.Lock) def check(url): host = urlsplit(url).netloc with host_locks[host]: # one request at a time per host for method in ("HEAD", "GET"): try: r = session.request(method, url, timeout=15, allow_redirects=True) if r.status_code < 400 or method == "GET": return url, r.status_code except requests.RequestException as e: if method == "GET": return url, type(e).__name__ unique = sorted({u for urls in data.values() for u in urls}) with cf.ThreadPoolExecutor(max_workers=16) as pool: results = dict(pool.map(check, unique)) json.dump(results, open("results.json", "w"), indent=2)

Checking unique URLs once (rather than once per repo) matters: badge URLs from the same service often appear in dozens of READMEs.

Step 4: score each repo

import json data = json.load(open("links.json")) results = json.load(open("results.json")) def broken(status): return not isinstance(status, int) or status in (404, 410) or status >= 500 rows = [] for repo, urls in data.items(): if not urls: continue bad = [u for u in urls if broken(results.get(u))] rows.append((len(bad) / len(urls), len(bad), len(urls), repo, bad)) for share, nbad, total, repo, bad in sorted(rows, reverse=True): print(f"{share:6.0%} {nbad:3}/{total:<3} {repo}")

Notice what's not counted as broken: 401, 403 and 429. Those usually mean "this site doesn't want to talk to a script", not "this page is gone". Count them separately as "couldn't verify", or you'll overstate the problem. Treat connection errors with some caution too, and re-run anything suspicious once before concluding it's dead.

Step 5: look for patterns

Once you have the table, the interesting part is why links broke. Group the broken URLs by host:

from collections import Counter from urllib.parse import urlsplit bad_hosts = Counter(urlsplit(u).netloc for u, s in results.items() if broken(s)) for host, n in bad_hosts.most_common(15): print(n, host)

Typical patterns worth looking for:

Retired badge and CI services. A single dead service can break badges across every repo you own.
Free app hosts. Demo links to free tiers that were later discontinued or put to sleep.
Your own old domains. Personal sites and project domains that lapsed.
Moved documentation. A docs site that changed its URL structure.
Branch names. Links to /blob/master/... in repos that renamed their default branch. GitHub redirects many of these, but not every pattern, so check yours.

Interpreting your number

Some questions to ask about your results:

Is rot concentrated in old repos? Plot broken share against repo creation date (available from gh repo list --json createdAt). If older READMEs are worse, that's link rot. If a new repo is bad, that's a copy-paste template problem.
Is it one host or many? One dead host means a quick find-and-replace across repos. Many hosts means ongoing rot that needs a recurring check.
Do the broken links matter? A dead badge is cosmetic. A dead "Getting started" link loses users.

Fixing it efficiently

Fix the top hosts first; they give the biggest win per edit.
Remove badges for services you no longer use.
Replace links to moved docs with their new locations, or link to an archived copy if the content is truly gone.
Consider adding a scheduled link check workflow to repos that matter, so the next rot gets caught early.

If you'd rather not maintain the scripts, linkbruno.com checks links on a schedule and tells you when one stops working.