A one-line dedupe rule was quietly deleting one in six real job postings from my dataset. Here is how it showed up, and the field I forgot to put in the key.

The symptom was a number I was proud of

After cleaning up a date bug, I looked at my pipeline's own funnel:

18,833 postings scraped → 15,548 after dedupe

A 17.4% duplicate rate. I had been treating that as proof the sources were messy and my dedupe was doing heroic work. It is not. On real job-board data, cross-source duplicates sit around 1–3%. Anything much above that is usually not the data being dirty — it is your key being too coarse.

So I stopped celebrating and measured what was actually colliding.

Measuring the collision

I took seven companies (stripe, gitlab, anthropic, figma, brex, ramp, openai), 2,835 postings, grouped them by my dedupe key, and then asked a much more useful question: inside each collision group, how many distinct locations are there?

The key was:

key = (job["company"].lower(), title_key(job["title"]))

The answer:

  • 194 collision groups
  • 188 of them (97%) contained more than one location
  • 353 rows merged away, 347 of which were distinct jobs

Real examples, straight from the run:

Company Title Locations collapsed into one row
stripe abuse investigator Dublin and Seattle, San Francisco, New York City
stripe account executive commercial hunter Chicago and Singapore
stripe account executive platforms hunter Chicago, Chicago, Illinois, London

Those are not duplicates. They are separate requisitions, with separate links and usually separate pay ranges. My "clean" dataset was throwing them away.

The fix is one field 

def location_key(location): text = clean_text(location or "").lower() text = re.sub(r"[,/|;]+", " ", text) text = re.sub(r"[-–—]+", " ", text) # US-Chicago == us chicago  return re.sub(r"\s+", " ", text).strip() key = (job["company"].lower(), title_key(job["title"]), location_key(job.get("location")))

Two details that bit me:

  1. Normalise the hyphen. Without it, US-Chicago and us chicago are different keys. My first test failure was exactly this.
  2. Fix the fingerprint too. If you build "only return jobs I have not seen" on an id derived from company+title, a second city for the same title looks like an already-seen job and never gets delivered.

What it changed

Measure Title-only key With location
7 companies: scraped → deduped 2,835 → 2,482 (12.5% lost) 2,835 → 2,824 (0.4% lost)
Full pool: scraped → deduped 18,833 → 15,548 (17.4% lost) 18,833 → 18,567 (1.4% lost)
Postings that state a salary 6,833 7,984 (+16.9%)

The recovered rows are mostly mid-to-low pay, so the median moved down slightly: $204,750 → $200,000. That is also a lesson — the bug was biasing the dataset upward, because the rows it deleted were disproportionately the many-city, lower-level sales and support reqs. If you only ever look at the median, the bias is invisible.

The rule I now follow

Treat the dedupe rate as an instrument, not a trophy. If it is above a few percent, print the collision groups and look at what differs inside them — location, department, employment type, seniority. The first differing field is almost always the one you forgot in the key.

And the inverse check is just as useful: if your dedupe rate is near zero, you are probably not deduplicating anything at all, and your cross-source duplicates are still in there.

The corrected numbers are live at radarlabs.surge.sh/salary-data.html — 18,567 postings, 7,984 with published pay, 259 companies, free CC BY 4.0 CSV. The age distribution is on a separate page if you came here for the earlier date-field bug. If you would rather not maintain the pipeline, the same data runs as a paid Apify Actor — but read the free CSV first and decide whether it is worth $0.0015 a row.