Shopify blocked my scraper and it wasn't my User-Agent

I open-sourced a little shopping tool. It had a polite fetcher, respected robots.txt, sent a real User-Agent, and checking that the address was public before it dialled out.

It had never once fetched a product.

Artifacts Apparel FAILED HTTP 429 for https://artifactsapparel.co.uk/products.json?limit=250&page=1 You Know Who's FAILED HTTP 429 for https://youknowwhos.co.uk/products.json?limit=250&page=1 Goodhood FAILED HTTP 429 for https://goodhoodstore.com/collections/sale/products.json?limit=250&page=1 Represent FAILED HTTP 429 for https://representclo.com/products.json?limit=250&page=1 ... 0 products checked, 0 in stock in your size 

Twenty shops, twenty-nines, zero products. And here is the part that made it interesting: the tool never failed loudly. It printed a table. It exited 0. It wrote an empty digest and posted it to Discord like a perfectly normal Monday.

The test that took two minutes

My first guess was the User-Agent. I'd written ClothingLab/0.1 (+https://github.com/casareanderson/clothinglab) — identifying myself with a real URL, the polite thing to do. Maybe Shopify didn't like the URL in it. Maybe they wanted a browser.

So I ran the same URL with the same User-Agent through two different clients:

import urllib.request, requests URL = "https://artifactsapparel.co.uk/products.json?limit=5&page=1" UA = "ClothingLab/0.1 (+https://github.com/casareanderson/clothinglab)" urllib.request.urlopen(urllib.request.Request(URL, headers={"User-Agent": UA})) # HTTPError: HTTP Error 429 requests.get(URL, headers={"User-Agent": UA}) # <Response [200]> 

Same URL, same User-Agent, same second. The User-Agent was never the variable. The HTTP client was.

What Shopify was actually looking at

requests and urllib look like the same request on paper. They are not:

Shopify's edge sits behind a WAF with bot rules that key on the shape of the request, not on what it claims to be. A urllib request announces itself with Connection: close and refuses compression. A WAF rule written to spot "a script that isn't a browser" catches it perfectly — and 429 Too Many Requests is a much softer answer than 403 Forbidden, which is what you'd get if it thought you were up to no good.

The thing that had me staring at it for a while: the tool did have a browser-ish User-Agent option. I had tried it. It still returned 429. Because I had been testing the wrong variable.

The fix

One import, one line:

-import urllib.request +import requests 

and the fetch became:

r = requests.get(url, timeout=self.timeout, headers={ "User-Agent": self.ua, "Accept": "application/json, text/html;q=0.9"}) if r.status_code != 200: raise FetchError(f"HTTP {r.status_code} for {url}") 

I kept the robots.txt check, the per-host delay, the public-address check and the User-Agent you configure — all of that was right. I just swapped the thing that was silently wrong.

Then I wrote the regression test, because "it works now" is not the same as "it works for a reason":

def test_fetcher_uses_requests_not_urllib(monkeypatch): """Shopify answers urllib with 429 and requests with 200 — the client is the difference, not the UA.""" seen = {} def fake_get(url, timeout=None, headers=None): seen["headers"] = headers return _Resp(200, b"hello") monkeypatch.setattr(fetch.requests, "get", fake_get) f = Fetcher(user_agent="ClothingLab/test") assert f._raw("https://example.com/products.json") == b"hello" assert seen["headers"]["User-Agent"] == "ClothingLab/test" 

The comment is the important part. Six months from now, someone will decide urllib is tidier and "it's stdlib, why not" — and the comment says exactly why not, in the place they'll be looking.

What it actually taught me

The bug wasn't clever. It was the most ordinary kind of bug there is: a program that runs fine and produces nothing.

I had a health table, a status line, a summary line, a digest, a Discord post and a schedule. Every one of them worked. The one number that mattered — products checked — was zero, and it was zero quietly. A green checkmark at the end of a successful run is the most dangerous thing a tool can produce, because it tells you the run happened, not that the work did.

Two habits I'd keep from this:

1. Test the wire, not just the logic. Every unit test I'd written passed, because they all stubbed the network. The bug lived in the eleven lines between "I have a URL" and "I have a response", and no unit test could ever have caught it. The two-minute test that fixed it was the first thing in the project that touched a real socket.
2. Make "zero" impossible to mistake for "fine". It costs nothing to have the tool refuse to post a digest that checked zero products from twenty sources, and say why. Silence and zero are not the same outcome, and a tool that cannot tell you the difference will take your data quietly.

The wider version of that lesson: when something is wrong in a way that produces a plausible result — zero products, no findings, no alerts, no changes — the first thing to check is the plumbing, not the content. A scraper that fetches nothing looks exactly like a scraper that found nothing.

And the specific version, which cost me an afternoon: if every single source fails the same way, that's a signal about your client, not about the sources.

🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.

🌐 More field notes: https://iamtech.cn1-lab.uk