I had a cron job that was supposed to finish in 2 minutes. It was still "running" 9 hours later. No error, no log line, nothing. The cause was one requests.get(url) call to a server that accepted the connection and then never answered.
The surprising part: this is documented behavior. The requests library has no default timeout. If you do not set one, the call can wait indefinitely.
The one-line fix
import requests r = requests.get("https://example.com/api", timeout=10) Now the call raises requests.exceptions.Timeout instead of hanging.
Connect timeout vs read timeout
timeout accepts a tuple: (connect, read).
r = requests.get(url, timeout=(3.05, 10)) - connect: how long to wait to establish the TCP connection.
- read: how long to wait between bytes from the server.
Careful: the read timeout is not a total deadline. A server that sends one byte every 9 seconds will never trip a 10-second read timeout. If you need a hard cap on total time, track it yourself:
import time, requests def get_with_deadline(url, total=30): start = time.monotonic() with requests.get(url, stream=True, timeout=(3.05, 10)) as r: chunks = [] for chunk in r.iter_content(8192): chunks.append(chunk) if time.monotonic() - start > total: raise TimeoutError(f"exceeded {total}s total") return b"".join(chunks) Stop repeating yourself: set it once on a Session
Typing timeout= on every call is how one gets forgotten. A small adapter applies a default to every request on a session, and urllib3's Retry handles transient errors:
import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry class TimeoutAdapter(HTTPAdapter): def __init__(self, *args, timeout=(3.05, 10), **kwargs): self.timeout = timeout super().__init__(*args, **kwargs) def send(self, request, **kwargs): if kwargs.get("timeout") is None: kwargs["timeout"] = self.timeout return super().send(request, **kwargs) def make_session(): retry = Retry( total=3, backoff_factor=0.5, # 0.5s, 1s, 2s between tries status_forcelist=[429, 500, 502, 503, 504], allowed_methods=["GET", "HEAD"], # only retry safe methods ) s = requests.Session() adapter = TimeoutAdapter(max_retries=retry) s.mount("https://", adapter) s.mount("http://", adapter) return s session = make_session() r = session.get("https://example.com/api") # timeout applied automatically Two details worth knowing:
-
allowed_methodsmatters. Retrying a POST can create duplicate records, so the example only retries GET and HEAD. -
Retryhonors theRetry-Afterheader on 429/503 responses by default, so you back off when the server tells you to.
Quick checklist
- Every network call has a timeout. No exceptions.
- Use a
(connect, read)tuple, with connect short. - Need a total deadline? Track elapsed time while streaming.
- Retry only idempotent methods.
Sources
- Requests docs, Timeouts: https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
- urllib3
Retryreference: https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry
What is the longest a "temporary" hang has gone unnoticed in one of your scripts, and how did you finally find it?
