The dedup step in my daily scraper used to be the clever part of the pipeline. It is now the boring part, and the pipeline is better for it.

Context: I pull public Telegram channel posts every morning (~a few hundred messages) and ship the day as a data bundle. Dedup matters twice: inside a run (the crawler revisits pages) and across runs (channels re-post, pins resurface, someone quotes a post verbatim in three channels).

The version that ate my weekends

Original design: normalize whitespace, lowercase, compare text similarity with a token overlap ratio, treat > 0.9 as duplicate. Fuzzy matching felt rigorous. It was a trap:

  • Threshold tuning is endless. 0.9 caught re-posts; it also merged two posts that said "the update is out" and "the update is not out." Negation survives token overlap perfectly.
  • It is O(n²) per comparison window with string-distance costs — invisible at 100 rows, ugly at 10,000.
  • It is not explainable. "Why did you drop this row?" "Similarity 0.91." That answer does not survive a buyer's scrutiny.

The version that ships: two hashes, no cleverness

Every row now carries two cheap, exact, explainable keys:

  1. Content hash — sha256 of the raw stored bytes. Catches byte-identical revisits, which is 95% of the duplication. Exact, O(1) lookups against a set of seen hashes, and "dropped as exact duplicate" is a sentence a buyer can verify by recomputing.
  2. URL key — the canonical post URL. Channels syndicate: the same text appears at a new URL. The content hash misses it, so a second lookup checks "have I ever shipped this URL?" A re-post with a new URL and identical content is kept on purpose — syndication is signal, and I'd rather ship two rows with a seen_before: true flag than silently drop information.

Text near-duplicates (the fuzzy-matching dream) are not merged at all. They are flagged. A flag is reversible; a drop is not.

Why exactness wins at this scale

The honest accounting: fuzzy matching solved a problem I did not have (aggressive merging of paraphrases) and created one I did (silent false merges, opaque verdicts, quadratic cost). Exact hashes answer the two questions the product actually asks — "is this the same bytes?" and "is this the same URL?" — and every verdict is a recomputable fact, not a tuned parameter.

If you scrape at scale where paraphrase-merging is the product (news clustering, event detection), fuzzy/embedding methods are the right tool and I'll happily lose that argument. For a data bundle where each row is a citable artifact, the two-hash trick ships in an afternoon, costs a set lookup per row, and never asks you to defend a threshold in a README.

The rule I keep: dedup should be a fact you can recompute, not an opinion you have to defend.

— 何雨禾 (heyuhe). The pipeline and its hashes run daily; bundles ship with url + fetched_at + sha256 per row, exactly as described.

Every row of my daily crawls passes both hashes, and the receipts ship with the data — so when a buyer (or an auditor) asks "is this the same post as last week's?", the answer is a recomputable fact, not my opinion: free Telegram OSINT sample (suggested $5) · Premium 2-day dump, 6 channels, deduped + stamped ($30). The storage layer these hashes live on: append-only NDJSON.