Anyone who has written a scraper knows the feeling. It works perfectly for three weeks, then one morning it returns nothing because the site renamed a CSS class or moved a div.
These are the habits I have picked up so far:
- Look for a JSON source first. A lot of sites ship their data as embedded JSON (Next.js
__NEXT_DATA__, JSON-LD, or an internal API the page calls). That changes far less often than the HTML around it. - Fail loudly. An empty result should be treated as an error, not a success. A scraper that quietly returns zero rows is worse than one that crashes.
- Run a tiny daily test. One known input with a check that the output still has the fields I expect. It catches breakage before anyone else does.
- Avoid deep selectors.
div > div:nth-child(3) > spanbreaks the moment anything shifts. Text labels and data attributes hold up better.
I am curious what everyone else does:
- What is your go-to trick for making a scraper last?
- Do you monitor them somehow, or just wait until something looks wrong?
- Has anyone had good results letting an LLM parse the page instead of writing selectors? I keep going back and forth on whether it is worth the cost and the occasional made-up field.
Would love to hear what has worked (and what has burned you).