Residential Proxies Are Getting Blocked? Diagnose the Scraper Before You Upgrade

Your scraper worked for months. Now it returns 429 Too Many Requests, a challenge page, or empty product data.

The tempting response is to buy a more expensive proxy pool.

That may help when the exit IP is the actual problem. But a proxy cannot fix every failure. A scraper can also be too bursty, lose session state between requests, misread an error page as valid content, or rely on a page structure that has changed.

Before changing providers, measure what is failing.

In this post, we’ll run a small experiment against a local mock website. It simulates a simple rate limit and session cookie. We’ll compare:

1. A burst of requests with no delay.
2. Requests spaced according to the server’s Retry-After response.

The example uses only Python’s standard library, so it runs without credentials, third-party packages, or external targets. Python documents urllib.request as a module for opening URLs and http.server as a basic server implementation; the latter is intended for simple use, not production deployment (urllib.request, http.server).

Scope: This is a diagnostic exercise for a local service you control. For real sites, use an official API or other permitted access method, follow the site’s published rules, and stop when asked to stop. The mock server does not reproduce any particular anti-bot product.

What we’re measuring

The mock site allows one request every 250 milliseconds per session. Faster requests receive a 429 response with a Retry-After header.

We’ll record:

• successful responses (200)
• rate-limited responses (429)
• total elapsed time
• number of returned product records

We are not measuring proxy quality, fingerprinting, or how to defeat a challenge. The point is to check whether a basic scheduling and session-state issue can explain a failure before adding infrastructure.

1. Create the mock website

Save this as mock_site.py:

from http.cookies import SimpleCookie from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from time import monotonic from uuid import uuid4 MIN_INTERVAL_SECONDS = 0.25 # In-memory state is enough for this local experiment. last_request_by_session = {} class Handler(BaseHTTPRequestHandler): def do_GET(self): cookie = SimpleCookie() cookie.load(self.headers.get("Cookie", "")) if "session_id" in cookie: session_id = cookie["session_id"].value else: session_id = uuid4().hex now = monotonic() previous = last_request_by_session.get(session_id) if previous is not None: elapsed = now - previous if elapsed < MIN_INTERVAL_SECONDS: retry_after = MIN_INTERVAL_SECONDS - elapsed self.send_response(429) self.send_header("Content-Type", "text/plain; charset=utf-8") self.send_header("Retry-After", f"{retry_after:.2f}") self.send_header("Set-Cookie", f"session_id={session_id}; Path=/") self.end_headers() self.wfile.write(b"Rate limited. Please retry later.") return last_request_by_session[session_id] = now body = b"""<!doctype html> <html> <body> <ul> <li class="product" data-id="101">Notebook</li> <li class="product" data-id="102">Desk lamp</li> <li class="product" data-id="103">Water bottle</li> </ul> </body> </html> """ self.send_response(200) self.send_header("Content-Type", "text/html; charset=utf-8") self.send_header("Set-Cookie", f"session_id={session_id}; Path=/") self.send_header("Content-Length", str(len(body))) self.end_headers() self.wfile.write(body) def log_message(self, format, *args): # Keep the demo output focused on the client results.  pass if __name__ == "__main__": server = ThreadingHTTPServer(("127.0.0.1", 8000), Handler) print("Mock site listening on http://127.0.0.1:8000") print("Press Ctrl+C to stop.") try: server.serve_forever() except KeyboardInterrupt: print("\nStopping server.") server.server_close() 

Start the server in a terminal:

python mock_site.py 

The server keeps request timestamps in memory and identifies a client using a cookie. The cookie is not an authentication mechanism; it simply gives the demo a stable session key.

2. Create the experiment

Save this as experiment.py in the same directory:

import re import time from collections import Counter from http.cookiejar import CookieJar from urllib.error import HTTPError, URLError from urllib.request import HTTPCookieProcessor, Request, build_opener URL = "http://127.0.0.1:8000/" REQUESTS_PER_RUN = 10 def make_opener(): # CookieJar preserves the mock site's session cookie between requests.  return build_opener(HTTPCookieProcessor(CookieJar())) def fetch(opener): request = Request( URL, headers={"User-Agent": "local-diagnostic-demo/1.0"}, ) try: with opener.open(request, timeout=3) as response: body = response.read().decode("utf-8") return response.status, body, None except HTTPError as error: # HTTPError also carries response headers, including Retry-After.  return error.code, error.read().decode("utf-8"), error.headers except URLError as error: raise RuntimeError(f"Could not reach the mock server: {error}") from error def count_products(html): return len(re.findall(r'class="product"', html)) def run(label, delay_seconds): opener = make_opener() statuses = Counter() product_count = 0 start = time.perf_counter() for index in range(REQUESTS_PER_RUN): status, body, headers = fetch(opener) statuses[status] += 1 product_count += count_products(body) print(f"{label}: request={index + 1} status={status}") if delay_seconds is not None and index < REQUESTS_PER_RUN - 1: time.sleep(delay_seconds) elapsed = time.perf_counter() - start return { "scenario": label, "requests": REQUESTS_PER_RUN, "200": statuses[200], "429": statuses[429], "products_returned": product_count, "elapsed_seconds": round(elapsed, 3), } if __name__ == "__main__": # Scenario A sends requests back-to-back.  burst = run("burst", delay_seconds=None) # Reset the server between scenarios so the first request starts cleanly.  print("\nRestart mock_site.py before the paced run.") print("Then run: python experiment.py --paced") print("\nFirst-run summary:") print(burst) # Optional second mode is selected in the command-line version below. 

The first script version runs only the burst case. To make it easy to run either scenario cleanly, replace the final if __name__ == "__main__": block with this one:

if __name__ == "__main__": import sys if len(sys.argv) != 2 or sys.argv[1] not in {"burst", "paced"}: raise SystemExit("Usage: python experiment.py burst|paced") scenario = sys.argv[1] if scenario == "burst": result = run("burst", delay_seconds=None) else: # The mock site's minimum interval is 0.25 seconds.  # 0.30 adds a small margin for scheduling jitter.  result = run("paced", delay_seconds=0.30) print("\nSummary") for key, value in result.items(): print(f"{key}: {value}") 

Run each scenario against a fresh server process. Restarting clears its in-memory rate-limit state.

Scenario A: burst

In terminal one:

python mock_site.py 

In terminal two:

python experiment.py burst 

Scenario B: paced

Stop and restart the mock server, then run:

python experiment.py paced 

3. Expected results

The exact runtime varies by computer. The response counts should be stable:

Why does the burst run return three product records? The first request succeeds and returns the page. The remaining nine get a plain-text rate-limit response, so the HTML parser finds no product rows.

The paced run takes roughly 2.7 seconds plus local request overhead: nine 0.30-second waits separate ten requests. That is slower in wall-clock time than sending a burst, but it returns ten times as many successful pages in this specific experiment.

This is a deliberately simple model. Real services may rate-limit by account, IP, route, time window, or combinations of signals. A 429 is a useful clue, not proof that a proxy is the problem.

A useful improvement: honor Retry-After

Fixed delays are easy to understand, but a client can use the server’s suggested wait time instead. Add this helper to experiment.py:

def retry_after_seconds(headers, default=1.0): if headers is None: return default value = headers.get("Retry-After") if value is None: return default try: return max(0.0, float(value)) except ValueError: # Retry-After may also be an HTTP date.  # This small demo handles only numeric seconds.  return default 

A production-ready client should support both forms of Retry-After, set a maximum wait, use a bounded retry count, and apply exponential backoff with jitter when appropriate. It should also avoid retrying indefinitely or increasing request volume when the target is already rejecting requests.

For this local mock, the cleanest fix is to schedule requests at a rate the service accepts. The server is under our control, so the test client could also be changed to retry after 429; we use a fixed 300 ms interval here to keep the comparison predictable.

Diagnosing a real scraper failure

Before buying a different proxy plan, collect a small, safe sample from a source you are authorized to access. Log the following for each request:

timestamp, URL path, status code, response content type, response byte count, elapsed milliseconds, retry count 

Avoid logging passwords, cookies, access tokens, or personal data. If you need to correlate requests, use a short-lived request ID rather than storing session secrets.

Then use the symptoms to narrow the cause:

1. Check the response before parsing it

A challenge page can return 200 OK. A parser that only checks the status code may silently save an HTML error page as if it contained product data.

Check the content type, expected page markers, and required fields. Track valid records per request, not only HTTP success.

2. Separate rate limits from transport failures

Record status codes and exception types separately. A 429 suggests the server is asking the client to slow down. Timeouts, DNS failures, connection resets, and 5xx responses point to different failure classes.

Do not convert all of them into “proxy failed.” That label hides useful evidence.

3. Test session continuity

Some workflows require state to persist across a sequence of pages. Confirm whether cookies or other documented session state are being retained when the source expects them.

Keep one session stable for a workflow when that is the intended behavior. Avoid rotating identity in the middle of a multi-step process; it can make the workflow less reliable and can violate the service’s expectations.

4. Check whether the page is the right data source

If the data is available through an official API, export, feed, or licensed provider, those are usually easier to maintain than parsing a user-facing page. If the page depends on JavaScript, first determine whether the data is available through a documented endpoint or a permitted integration.

5. Change one variable at a time

A useful troubleshooting sequence is:

1. Run a low-volume baseline.
2. Add a conservative delay.
3. Preserve expected session state.
4. Validate response content and extracted fields.
5. Only then compare network paths or proxy options, if the use case permits them.

If you change the delay, proxy provider, browser, headers, and parser together, you may improve the outcome without learning why.

When does a proxy change make sense?

A proxy is a network route. It may be relevant when the authorized service requires a particular region, your organization uses a controlled egress path, or you have evidence that the current exit route is the failing component.

It is less useful when:

• the server is returning 429 because requests arrive too quickly;
• the client drops required cookies between pages;
• the parser is using outdated selectors;
• the page is an error or challenge response being counted as valid data;
• the source offers an API that better fits the task.

When evaluating a provider, compare more than the advertised IP count. Test the same approved workload and measure:

• valid records per request;
• success rate over a defined sample;
• latency at the rate you actually need;
• total cost per valid record;
• session stability for workflows that require it;
• support for the required geography and protocol.

A cheaper plan with a lower cost per gigabyte may still cost more per usable record if it has higher failure rates or requires more retries.

Limits of this experiment

The mock site has one fixed threshold, one in-memory process, and one cookie-based session model. It does not model distributed rate limiting, changing page markup, authentication, browser rendering, or any commercial anti-bot service.

That limitation is intentional: the code isolates one variable and makes the outcome reproducible. It shows that a scheduling problem can look like an infrastructure problem. It does not show that delays or cookies will resolve blocks on a real site.

Use the same measurement approach on systems you own or are authorized to test, and treat each site’s published access rules as part of the technical requirements.

Takeaway

When a scraper starts failing, measure before you upgrade:

• inspect status codes and response bodies;
• count valid extracted records, not just successful connections;
• preserve session state when the workflow requires it;
• use server-provided retry guidance and bounded backoff;
• compare proxy options only after you have evidence that the network path is the bottleneck.

That gives you a better diagnosis—and a more honest cost comparison—than choosing a proxy tier based on guesswork.