A price comparison site is only as useful as its last check. A deal that was real on Monday can be gone by Friday, and a price that moved overnight makes yesterday's guide wrong. At Savlume, we publish buying guides that list current prices, so keeping those prices fresh is a core engineering problem, not an afterthought. This post explains how we think about incremental crawling and change detection — the pipeline that decides what to re-check, when, and how to react when a price moves.
Why you cannot just re-crawl everything
The naive approach is simple: crawl every product page on a schedule and rewrite the database. It works at small scale and falls apart quickly. Retailer pages are heavy, rate limits are real, and most prices have not changed since the last check. Re-crawling everything wastes bandwidth, invites blocks, and buries the few real changes in a pile of identical snapshots.
The fix is to crawl incrementally: only fetch what is likely to have changed, and only act when the fetched data actually differs from what we already have.
Prioritising what to check
Not every product deserves the same check frequency. We rank products by a few signals:
- Volatility. Products whose prices moved recently get checked more often. A stable price that has not moved in a month can wait.
- Traffic. Guides that readers open every day get fresher data than guides nobody has visited in weeks.
- Seasonality. Around Black Friday and other sale events, everything gets checked more often, because that is when prices move fastest.
- Deal status. A product currently listed as a deal — a price below its usual level — gets checked daily, because a deal ending is exactly the change readers care about most.
This is a priority queue, not a fixed schedule. The crawler works through the highest-priority items first, so a limited crawl budget still covers the prices that matter.
Change detection: compare, don't just store
Fetching a page is only half the job. The other half is deciding whether anything changed. We keep the last known snapshot for each product — price, list price, availability, and the verification status — and compare the new fetch against it field by field.
Three outcomes matter:
- No change. The price is the same. We update the "last checked" timestamp and move on. Nothing is republished, and no alert fires.
- Price changed. The new price replaces the old one, the change is logged with a timestamp, and the guide's verification status is re-evaluated. A price drop can become a new deal; a price rise can end one.
- Page or structure changed. The price field is missing or the page layout shifted. This is not a price change — it is a signal that the extractor may be broken, and it goes to a review queue instead of silently overwriting good data.
That third case is the one most small crawlers miss. A missing price should never be treated as a zero price or a removed product. It means "we don't know", and the guide should keep showing the last verified price with its check date rather than publishing a guess.
Verification status: retailer-verified vs. source-reported
At Savlume, every price carries a verification label. A price is retailer-verified when we confirmed it directly on the retailer's own page; it is source-reported when it comes from a secondary source and has not been confirmed at the retailer. Change detection feeds directly into this: a retailer-verified price that we re-confirm stays retailer-verified with a fresh timestamp. A price we can no longer confirm at the retailer drops back to source-reported until it is verified again.
This matters because a guide is a promise. Readers should be able to tell, at a glance, which prices we actually saw at the store and which ones we are passing along. The labels are part of the data model, not decoration on the page.
Publishing the change, fast
A detected price change is only useful if it reaches readers. Our guides are served through a small API surface — /api/v1/items for structured product data and /feed/all.xml for an RSS feed — and an MCP endpoint at /api/mcp plus an llms.txt file so AI agents can read the same data. When a price changes, the guide page, the API, and the feed all reflect the new value from the same source of truth, so there is no window where the guide says one price and the feed says another.
The RSS feed is the fastest path to readers who follow it: a price drop that the crawler confirms in the morning can be in a subscriber's reader before lunch, without anyone manually editing a page.
What we learned
- Check the prices readers actually see. A crawler that checks everything equally will always be behind on the pages that matter.
- Treat "unknown" as a state, not an error. A failed fetch should never overwrite a good price with a blank.
- Log every change with a timestamp. Price history is what turns "this is cheap now" into "this is the cheapest it has been in a month", and that history only exists if you record it.
- Keep one source of truth. The guide page, the API, and the feed should all read from the same data, so a change propagates everywhere at once.
Price tracking is not a scraping problem. It is a freshness problem: knowing what changed, deciding what it means, and getting the new answer to readers before it goes stale again.
Prices in Savlume guides are marked retailer-verified or source-reported, and every guide links back to the retailer so you can check the current price before you buy. Browse the guides at https://savlume.com/
Follow Savlume Deals on WhatsApp: https://whatsapp.com/channel/0029VbE9P7J7DAX7rD8TAC2U
