I run a Discord bot that searches second-hand marketplaces across Europe. Some of those sites block datacentre IPs, so part of the traffic goes through a residential proxy, billed per gigabyte.

Last week I sat down to work out what that actually costs. Three things turned out to be wrong, and the most expensive one was my own measurement.

The number that mattered wasn't per customer

I had assumed proxy traffic scaled with users. It doesn't. It scales with standing alerts, because an alert is a loop that runs forever whether anyone is awake or not.

A scan cost about 3 MB. The alert loop ran every 10 minutes. So:
One alert. A customer on the top tier can have four. On a 10 GB bundle I couldn't afford a single user before I had one.

That reframing matters more than any optimisation. Cost per customer is a number you can price around. Cost per unit of recurring background work is a number that grows while you sleep.

My counter was measuring nothing

I built a byte counter so I'd stop guessing. It read Content-Length off each response and added it up. After a full nightly run over 115 sources it reported 36 MB, which seemed suspiciously cheap.

It was. Most servers send HTML with Transfer-Encoding: chunked and no Content-Length header at all — so every one of those responses counted as zero. The heaviest pages, the ones I most wanted to measure, were exactly the ones being skipped.

I proved it with a local server that serves a chunked page:
Not a bit low. Zero.

The fix for the browser side was to ask Chromium directly, through CDP, which reports encodedDataLength — bytes on the wire, headers included, still compressed. That is exactly what the proxy bills for.

cdp = await page.context.new_cdp_session(page) await cdp.send("Network.enable") cdp.on("Network.loadingFinished", lambda e: tel(int(e.get("encodedDataLength") or 0)))

For plain requests, urllib3 already tracks it: response.raw.tell() gives bytes read off the connection, compressed. Falling back to len(response.content) overcounts by several times on HTML, which is still better than zero.

Three fixes, none of them clever

The browser was downloading every product image. It only ever reads the HTML. One route handler, and the heaviest pages stopped being heavy:

async def block(route): if route.request.resource_type in {"image", "media", "font"}: await route.abort() else: await route.continue_() await page.route("**/*", block)

JavaScript and CSS still load, so bot checks still pass.

Most sites don't need residential bandwidth at all. The proxy flag defaulted to True and 45 of 48 call sites had simply inherited it. Nobody ever chose. I tested each source four ways — requests and browser, with and without proxy — and five of ten didn't need it. One of those was the single most-used source in the whole product, running in 25 countries.

That one took measuring, not reasoning. Two of the ten genuinely do need the proxy, and I'd have guessed wrong about which.

Alerts checked every source every round. Now they rotate: a couple of sources per round instead of all of them, so each source comes round every 90 minutes instead of every 30. Three times less traffic.

The rotation turned into the one piece I'd call a design decision rather than a fix: how many sources you get per round depends on your plan. The top tier still gets everything every round and hears about new listings first. A cost problem became a line on the pricing page.

The part where I broke it

I also made the shared browser helper try a direct connection first and fall back to the proxy if the page looked unusable. "Unusable" meant: too small, or containing a known block phrase.

The next morning, eight fewer sources were delivering. One of them returned 51 product cards with no price on any of them — a page big enough and clean enough to pass my check, and useless anyway.

The check was testing the wrong thing. "Is this page large and not obviously blocked" is not the same question as "does this page contain what we came for", and the second one is per-source work I hadn't done. I put the whole behaviour behind a flag and turned it off, keeping only the five sources I had actually measured.

The nightly health check caught it, which is the only reason this is a paragraph and not an outage.

What I'd tell past me

Measure per unit of recurring work, not per customer. Verify the measurement before you trust it — mine was off by a factor I still can't name precisely, in the reassuring direction. And when you turn a default on for everything at once, make it a flag, because you will be turning it off at eight in the morning.

From roughly 13 GB per alert per month to under 1. No new infrastructure, no plan upgrade, and the only reason I found any of it is that I stopped estimating and started counting.