For a while our server logs made no sense. On a normal day we'd get over thirty thousand page views from more than twenty-one thousand different IP addresses. Every address made one or two requests and never came back. Look any of them up and you'd find exactly what you'd expect from a real person: a home broadband line in Brazil, a mobile network in Pakistan, a cable ISP in Mexico.
None of them were people. It was one scraper renting a residential proxy pool, and every request went out through somebody's home router or phone.
I run an IP lookup site, so "is this address a real household or a proxy?" is a question we get asked every day, and one we had to answer for ourselves. The short version: the address alone usually can't tell you. Here is what can, roughly in the order we check it.
1. Where the address claims to be versus where it can physically be
Light in fibre covers about 100 km per millisecond of round trip, give or take. So if a "residential IP in Zagreb" answers a ping from a probe in Frankfurt in 2 ms, it isn't in Zagreb. It is sitting in a data centre a few hundred kilometres from Frankfurt, wearing a Zagreb label.
We use that rule a lot, but carefully. RTT can rule a city out; it can almost never prove one. 3 ms can't tell a suburb from the city next to it, and we learned that the hard way when an automated pass "confirmed" a batch of suburbs as their neighbouring big city and we had to undo it. Now a row only moves when the ping makes the claimed place impossible, not merely unlikely.
2. Geofeeds that are too pretty
Network operators can publish a geofeed: a CSV that says which of their prefixes sits in which city. Honest ones are boring. A cable company's feed lists the towns it actually serves, in one country.
The fake ones look like a geography quiz. One feed we rejected mapped hundreds of /24s from a single block to small places in Denmark, Cambodia, Albania and Nigeria. Nobody has a broadband footprint like that. Somebody selling proxies does.
Even feeds that look sane get checked against the regional registries before we trust a country change. Out of 332 geofeed country claims that passed our other filters in one round, the registry agreed with exactly one. That number still surprises me.
3. Two databases agreeing is not proof
It's tempting to think that if two well-known IP databases put an address in the same country, it must be right. We tried that on 3,373 rows. When we pinged a sample, about one in five of the agreed locations was physically impossible. Both sources had fallen back to the same thing: the headquarters address in the registry record. A big carrier's HQ city ends up on addresses it uses on another continent.
4. Reverse DNS still says a lot
Real ISPs name their customer addresses, and they're boringly consistent about it. A home DSL line in Slovenia comes back as something like bsn-61-98-230.dynamic.siol.net. Pools of proxies on rented hosting space often have no reverse DNS at all, or names that clearly belong to a data centre while the geolocation claims a living room.
It's not proof on its own. Some small ISPs never set PTR records. But a "residential" range with no PTRs, announced by a hosting network, is a strong hint.
5. Who announces it, and who it's registered to
The organisation in the registry record and the network that actually announces the range in BGP should usually be the same, or obviously related. When a block registered to a home broadband company is announced by a small hosting AS, it's often been leased out, and leased ranges are where a lot of proxy and VPN capacity lives. We keep over 110,000 VPN and proxy ranges flagged in the database, and leased blocks are one of the most common places they turn up.
6. The part network data can't see
Here's the honest bit. The worst residential proxies are real home connections: an SDK in a free app or a "bandwidth sharing" program on someone's PC routes other people's traffic through their line. Location, ASN and reverse DNS are all genuinely residential, because the line is. No lookup will flag that address, and anyone who claims otherwise is guessing.
What gives those away is behaviour, not the address. The scraper I started with gave itself away in the request headers. Every request claimed to be desktop Chrome, yet none sent the Accept-Language header every real Chrome sends, and the Chrome client hints were missing or contradicted the user agent. A later fleet used ten fixed user agents with the same time zone and screen size on every request, spread over a long list of addresses. One address does nothing suspicious. Ten thousand addresses that share a fingerprint do.
So our labels try to stay honest: an address on a home ISP gets the ISP label, even when we suspect it's being rented out, and the proxy flag goes on ranges where the network evidence holds up.
What I'd check, if you're deciding about one address
- Does the location survive a ping from somewhere nearby?
- Does the reverse DNS look like that ISP's customers, or like a server?
- Is the announcing network the one the range is registered to?
- Is the address part of a block that's been carved into many countries?
- And if you have the traffic: do the headers look like the browser they claim to be?
You can run the first four on any address with our IP lookup. It shows the registry owner, the announcing network, reverse DNS and the network type side by side. The fifth one needs your own logs, and in my experience that's where the interesting answers are.
