Anyone who has written a scraper knows the feeling. It works perfectly for three weeks, then one morning it returns nothing because the site renamed a CSS class or moved a div.

These are the habits I have picked up so far:

  • Look for a JSON source first. A lot of sites ship their data as embedded JSON (Next.js __NEXT_DATA__, JSON-LD, or an internal API the page calls). That changes far less often than the HTML around it.
  • Fail loudly. An empty result should be treated as an error, not a success. A scraper that quietly returns zero rows is worse than one that crashes.
  • Run a tiny daily test. One known input with a check that the output still has the fields I expect. It catches breakage before anyone else does.
  • Avoid deep selectors. div > div:nth-child(3) > span breaks the moment anything shifts. Text labels and data attributes hold up better.

I am curious what everyone else does:

  1. What is your go-to trick for making a scraper last?
  2. Do you monitor them somehow, or just wait until something looks wrong?
  3. Has anyone had good results letting an LLM parse the page instead of writing selectors? I keep going back and forth on whether it is worth the cost and the occasional made-up field.

Would love to hear what has worked (and what has burned you).