Your link checker's timeout is lying to you: a local benchmark of false positives

Every link checker has a timeout setting. Most people never touch it. That's a problem, because it quietly decides how many of your "broken links" are actually broken.

Set it too short, and slow-but-working pages get reported as dead. Set it too long, and one sluggish server can stall your whole run. Let's measure the trade-off with a local experiment you can reproduce on your own machine.

Important: everything here runs against a local test server with simulated response times. The numbers are illustrative of the shape of the trade-off, not measurements of the real web.

The setup

We'll create a fake "internet" on localhost: 200 URLs, each of which responds after a random delay drawn from a long-tailed distribution (most fast, some slow, a few very slow). A handful are genuinely broken and return 404 immediately.

# server.py import random, time from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler random.seed(7) DELAYS = {} for i in range(200): # lognormal: most responses fast, a long tail of slow ones  DELAYS[f"/page/{i}"] = min(random.lognormvariate(-1.2, 1.0), 30) BROKEN = {f"/page/{i}" for i in random.sample(range(200), 10)} class H(BaseHTTPRequestHandler): def do_GET(self): if self.path in BROKEN: self.send_response(404); self.end_headers(); return time.sleep(DELAYS.get(self.path, 0)) self.send_response(200); self.end_headers() self.wfile.write(b"ok") def log_message(self, *a): pass ThreadingHTTPServer(("127.0.0.1", 8799), H).serve_forever()

Start it in one terminal:

python3 server.py

The checker

A deliberately simple checker: concurrent GET requests with a configurable timeout. Anything that times out or returns 4xx/5xx is "broken".

# check.py import sys, time, concurrent.futures as cf import urllib.request, urllib.error URLS = [f"http://127.0.0.1:8799/page/{i}" for i in range(200)] def check(url, timeout): try: with urllib.request.urlopen(url, timeout=timeout) as r: return url, r.status except urllib.error.HTTPError as e: return url, e.code except Exception: return url, "timeout" def run(timeout, workers=20): t0 = time.time() with cf.ThreadPoolExecutor(workers) as ex: results = list(ex.map(lambda u: check(u, timeout), URLS)) elapsed = time.time() - t0 real_404 = sum(1 for _, s in results if s == 404) timeouts = sum(1 for _, s in results if s == "timeout") return elapsed, real_404, timeouts print(f"{'timeout':>8} {'404s':>5} {'timeouts':>9} {'reported broken':>16} {'run time':>9}") for t in (0.5, 1, 2, 5, 10, 30): elapsed, r404, to = run(t) print(f"{t:>7}s {r404:>5} {to:>9} {r404 + to:>16} {elapsed:>8.1f}s")

Run it in a second terminal:

python3 check.py

What you'll see

The exact figures depend on your machine and the random seed, but the pattern is consistent:

The 404 count stays at 10 across every timeout. Genuinely broken links are found regardless of the setting, because they fail fast.
The timeout count drops sharply as the timeout grows. On one run on our box: 51 timeouts at 0.5s, 20 at 1s, 8 at 2s and 0 from 5s up.
Run time grows with the timeout until it covers the slowest page, then flattens. Here it went from about 3 seconds to about 6.4, and stayed there from 5s on, because no simulated page was slower than that.

Here's that run in full:

 timeout 404s timeouts reported broken run time 0.5s 10 51 61 3.1s 1s 10 20 30 4.2s 2s 10 8 18 5.1s 5s 10 0 10 6.4s 10s 10 0 10 6.4s 30s 10 0 10 6.4s

(You may see BrokenPipeError noise in the server terminal. That's the server trying to answer requests the checker already gave up on, which is exactly the effect we're measuring.)

At a 0.5-second timeout, 61 links were "broken" and only 10 really were. So with short timeouts, most of your "broken links" report can be false positives. They aren't broken at all. They're just slow.

Why this matters in real reports

On the real web, slow responses come from many sources: overloaded small sites, servers far away, rate limiting kicking in because your checker sent too many requests, or cold starts on serverless hosting. None of these mean the link is dead. But a checker with an aggressive timeout reports them the same way as a 404.

The result is a report people stop trusting. If half the "broken" links work when someone clicks them, the real ones get ignored too.

What to do with this

1. Separate timeouts from errors in the report

A timeout and a 404 are different findings. A 404 is a fact. A timeout is "couldn't tell". Report them in separate groups so people fix the 404s first.

2. Retry timeouts once, later

Many slow responses are temporary. Re-check timed-out URLs at the end of the run, ideally with a longer timeout. Only report links that time out twice.

first = {u for u, s in results if s == "timeout"} retry = [check(u, timeout=15) for u in first] still_broken = [u for u, s in retry if s == "timeout"]

3. Pick a timeout based on what "too slow" means to you

If a page takes 20 seconds to respond, it's technically working but your readers will give up. Some teams treat "slower than 10 seconds" as a problem worth fixing even if it eventually loads. That's a valid choice, as long as the report labels it "slow" rather than "broken".

4. Don't let a few slow hosts block the run

With a long timeout, use enough concurrency (and per-host limits, to stay polite) so slow URLs don't hold up everything else.

Try it yourself

Change the distribution in server.py: make it more or less long-tailed, add more broken pages, simulate a server that slows down when it gets many requests at once. Then see which timeout and retry setting gives you the cleanest report in reasonable time.

If you'd rather not tune this yourself, linkbruno.com separates timeouts from hard errors and re-checks uncertain results before reporting them. But the experiment above is worth running once either way, just to see how much a single setting changes the story.