Residential Proxies Made My Lazada Scraper Worse: the Gate Reads the Handshake, Not the IP

I run a visa agency in Bali. The part of my week that has nothing to do with visas is building scrapers on Apify, and the one I rewrote most times this autumn reads Lazada — the marketplace that runs six separate storefronts in Southeast Asia: Indonesia, Malaysia, Singapore, the Philippines, Thailand and Vietnam, each with its own domain, currency and catalogue.

I expected the hard part to be the usual marketplace problem: pagination and messy prices. The hard part turned out to be a gate that does not care how much you pay for your IP addresses.

The 1.7 KB page that says 200

Lazada's anti-bot answer is not a 403. It is HTTP 200 with about 1.7 KB of HTML, and the only reliable way I found to recognise it is by its own markers:

export function isAntiBot(body) { return typeof body === 'string' && (body.includes('_____tmd_____') || body.includes('x5secdata')); } 

Before I checked for those strings, a blocked product page went through the parser as a valid response, the parser found no product in it, and the run wrote "product data not found" without ever retrying. The scraper was reporting a parsing failure for what was actually a block — the single most expensive bug in the Actor's history, because it looked like my code and was not.

The measurement that inverted my assumptions

On 24 September 2026 I ran the same request — page one of a Lazada search — eight times over each of four routes, from the Apify platform. Two HTTP clients: got-scraping with generated browser headers, and impit, which replays a real Chrome TLS fingerprint. One number per cell: how many of the eight attempts came back with products instead of the anti-bot page.

Read the columns, not the rows. Swapping the client turned a total failure into a near-total success on three of four routes. Swapping the IP for the most expensive option available made things worse — country-pinned residential was the only route that never got through at all.

That is the opposite of the folklore. The received wisdom is that datacenter IPs are burned and residential is the fix. For Lazada, in that hour, the gate was looking at the handshake first; the address only mattered after the handshake passed, and a residential exit in the marketplace's own country looked more suspicious than Apify's plain egress, not less.

The second thing I learned that afternoon is not to trust a single measurement, including mine. I re-ran the same matrix an hour later and the best route had dropped from 8 of 8 to 1 of 6. The gate is stateful: it flags addresses, and it cools down. Indonesia was consistently the strictest of the six marketplaces.

So the retry logic cannot be a fixed list of routes. It is a ladder that starts with the cheapest thing that works most often and alternates after that:

// attempt 0: platform egress. then alternate datacenter / residential. const tierFor = (attempt) => (attempt === 0 ? 'direct' : attempt % 2 ? 'std' : 'res'); 

Two guards make the ladder affordable. Each page gets a 120-second budget — past roughly two minutes, a hot gate is not going to open for rotation alone, and every further attempt is paid traffic. And a run-level breaker gives up entirely after three pages in a row exhaust their ladder:

const breaker = { streak: 0, open: false, ok() { this.streak = 0; }, fail() { if (++this.streak >= 3) this.open = true; }, }; 

A blocked answer is ~2 KB, so thirty attempts cost almost nothing in traffic — but thirty attempts on every page of a 5,000-product job costs time, and the run's own timeout is the resource that actually runs out. Whatever the run never reached goes into a NOT_ATTEMPTED record instead of silently disappearing from the dataset.

The JSON endpoint is guarded. The HTML holding the same JSON is not

Lazada's search results have a convenient JSON endpoint: add ?ajax=true to any search, category, shop or brand URL and you get the product list as data. Since September 2026 that endpoint answers with the anti-bot page for most sessions — while the ordinary HTML page still renders, and that page embeds the very same payload in a window.pageData = {...} assignment.

So the HTML page became the primary source and the JSON endpoint the fallback. Pulling the object out needs brace matching rather than a regex, because product titles on Lazada contain { and } often enough to break a naive count:

for (let i = start; i < html.length; i += 1) { const ch = html[i]; if (escaped) { escaped = false; continue; } if (ch === '\\') { escaped = true; continue; } if (ch === '"') { inString = !inString; continue; } if (inString) continue; if (ch === '{') depth += 1; else if (ch === '}' && --depth === 0) return JSON.parse(html.slice(start, i + 1)); } 

Reviews live somewhere else again — not on www, but on a sibling host: https://my.lazada.co.id/pdp/review/getReviewList?itemId=...&pageNo=1. Same for all six countries, with the country's own my. host.

totalResults is not a count

Each product card carries the search's claimed total. Eleven searches from my runs on 24 September 2026, seven keywords, all six marketplaces:

Nine of eleven searches reported exactly 4,080. A search page serves 40 products, so 4,080 is 102 pages' worth — a ceiling, not a measurement. And Malaysia is the tell: the same run got two different totals from two consecutive pages of one search, which no real count does.

Use it as "there is more than you asked for" and nothing else. If you need market-sized coverage, slice the query — by category, by brand page, by price band with minPrice/maxPrice — and dedupe on itemId. The 4,080 ceiling applies per query, not per catalogue.

soldQuantity has a related honesty problem: it is parsed from Lazada's own abbreviated string. "24.0K sold" becomes 24000, and the true number is somewhere in a 100-unit band. Below a thousand the counts are exact, above it they are three significant digits. I keep soldText next to the number so the rounding stays visible.

The product page does not know the price

The clean surprise in the schema. A search card and a full product record share only 12 keys; 22 are card-only and 12 are detail-only:

• Card only: price, priceText, originalPrice, discountPercent, rating, reviewCount, soldQuantity, location, isLazMall, badges, inStock, position, keyword…
• Detail only: description, descriptionHtml, highlights, variants, variantCount, imageCount, category, availability, sku…
• Shared: itemId, title, url, brand, sellerId, images, country, currency, marketplace, imageUrl, type, scrapedAt.

There is no price field on a product detail row at all. Price on Lazada belongs to a SKU, and a product page has several — one real iPhone 15 record came back with three variant combinations of colour and storage. So "scrape the product pages" is the wrong plan for a price tracker: the search card is where the price lives, and enrichProducts exists to merge the description and gallery into those cards rather than to replace them.

Page one of the reviews is not public opinion

70 reviews from two products, pulled with the default ordering: 66 of them five stars. Four fours. Zero ones. Default review ordering on Lazada is not chronological and not neutral, so averaging what the first pages give you produces a 4.9 for anything.

Three more things real review rows taught me:

• Dates come in two shapes in the same payload. 61 of 70 were absolute, in the local format — "13 Agt 2026", Indonesian month abbreviations. The other 9 were relative: "4 minggu lalu", "3 weeks ago". The relative ones can only be resolved at scrape time, so their ISO timestamps are approximate and carry a time of day. I keep both reviewTimeText and the derived reviewTime so nobody mistakes one for the other.
• The purchased variant is country-dependent. variant was filled on 50 of 50 Singapore reviews and 0 of 20 Indonesian ones, same field, same code path.
• Review text is partly a form. Lazada prompts buyers with phrases, and they survive into the data: one sentence about combination skin appeared in 3 of 20 Indonesian reviews of the same product, word for word. If you are running sentiment over this, dedupe sentences before you count anything.

Calling it

Two things worth knowing about the API: the country is a separate field from the URLs, and mixing countries in one run is fine because every row carries its own country and currency.

curl -X POST "https://api.apify.com/v2/acts/lergassy~lazada-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"mode":"search","country":"ID","searchTerms":["kopi bubuk"],"sortBy":"price_asc","maxItems":40}' 

The Actor is Lazada Scraper — pay-per-result, $0.002 per search card, $0.004 per full product record, $0.001 per review, no start fee, and blocked attempts are on me rather than on your bill. That pricing decision is downstream of everything above: when a gate's pass rate swings from 8 of 8 to 1 of 6 within an hour, charging per request would be charging for the weather.

When it is the wrong tool. It will not give you minute-by-minute price history — you schedule runs and build the history yourself. It cannot see anything behind a login: order data, your own seller dashboard, buyer contact details. And if you want one product's price watched continuously rather than a market read in bulk, a run that has to negotiate an anti-bot gate is a clumsy way to get one number; scrape the search card for the SKU you care about, on a schedule, and keep the row.

The general lesson I would take to the next marketplace: before you spend money on better IP addresses, measure whether the gate is even looking at them. Mine was looking at the handshake.