I wrote this article with AI assistance. I ran every snippet on 5 Nov 2026 with Node 24 and Python 3.13 (and checked them on Node 18 and Python 3.9), and all output is copied from those runs. The sample data is fictional.

If you scrape the same source more than once, you get duplicates. A second run overlaps the first. Two search queries return the same company. A directory and a map listing describe one business in two formats. Before you load the data anywhere, you need one row per real-world thing.

Deduplication (dedup) means finding the rows that describe the same thing and keeping one. Merging means combining those rows into one row that has the best value for each field.

Who this is for: developers who collect data from scrapers, APIs or CSV exports and need to combine several runs or sources.

What you need: Node 18 or later for the JavaScript script, or Python 3.9 or later for the Python scripts. None of them needs any packages.

The sample data

Save this as leads.json. It holds eight company records from two scraper runs, on 1 and 4 November. The .example domains are reserved for documentation, so none of these companies is real.

[ {"name": "Acme Sdn Bhd", "website": "https://www.acme.example/", "email": "Sales@Acme.example", "phone": "+60 3-1234 5678", "city": "", "scrapedAt": "2026-11-01"}, {"name": "Kopi Corner", "website": "http://kopicorner.example/menu?utm_source=maps", "email": "", "phone": "012-345 6789", "city": "Penang", "scrapedAt": "2026-11-01"}, {"name": "Batik House", "website": "", "email": "hello@batikhouse.example", "phone": "", "city": "Melaka", "scrapedAt": "2026-11-01"}, {"name": "ACME Sdn. Bhd.", "website": "acme.example", "email": "sales@acme.example ", "phone": "03-1234 5678", "city": "Kuala Lumpur", "scrapedAt": "2026-11-04"}, {"name": "Kopi Corner ", "website": "https://kopicorner.example/menu", "email": "hi@kopicorner.example", "phone": "+60 12-345 6789", "city": "George Town", "scrapedAt": "2026-11-04"}, {"name": "Batik House", "website": "https://batikhouse.example", "email": "", "phone": "06-222 3333", "city": "Melaka", "scrapedAt": "2026-11-04"}, {"name": "Nasi Lemak Co", "website": "https://nasilemak.example", "email": "order@gmail.com", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04"}, {"name": "Roti Studio", "website": "", "email": "rotistudio@gmail.com", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04"} ]

A person can see five companies here. Code can't yet. The two Acme rows differ in name, website, email and phone format. The two Batik House rows share no field except the name and city.

1. Choose a dedup key

A dedup key is the value that identifies a record. Two rows with the same key are duplicates, so choose the key first.

Good keys are stable and unique: an ID from the source (a place ID or a job ID), a product URL, or a company's website domain. Bad keys change between runs or are shared by unrelated records: names, page titles, timestamps, and the position of a row in the results.

For company data, the website domain is a good key. When a row has no website, the domain of a company email address is a fallback. Free email domains such as gmail.com are not a fallback, because thousands of unrelated companies share them. A row with no reliable key should stay as it is. Guessing would merge companies that only look alike.

2. Normalise the key, then deduplicate (JavaScript)

Normalising a value means rewriting it into one standard form, so that different spellings of the same thing compare as equal. dedupe.mjs applies these rules:

Value Rule Example
Any text Trim and collapse whitespace "Kopi Corner " → "Kopi Corner"
URL Drop http(s):// and www., lowercase the host, drop the trailing slash and tracking parameters (utm_*, fbclid, gclid), sort the remaining parameters http://www.Kopicorner.example/menu/?utm_source=maps&b=2&a=1 → kopicorner.example/menu?a=1&b=2
Email Trim and lowercase Sales@Acme.example → sales@acme.example
Phone Keep the digits, compare the last 9 +60 12-345 6789 and 012-345 6789 → 123456789

The phone rule ignores country codes and the leading trunk 0. That suits data from one country. If your data mixes countries, two different numbers can share their last 9 digits, so parse them with a library such as libphonenumber instead.

The script below builds a key for each row and removes duplicates in one of two ways:

  • Keep first keeps the first copy it sees. It is the simplest, and works in one pass over a stream.
  • Keep newest keeps the copy with the latest scrapedAt date, so fresh data replaces old data.

Save it as dedupe.mjs:

// dedupe.mjs - remove duplicate records from several scraper runs. // Node 18+. No dependencies. Usage: node dedupe.mjs leads.json import { readFileSync } from 'node:fs'; const FREE_MAIL = new Set(['gmail.com', 'yahoo.com', 'hotmail.com', 'outlook.com']); const TRACKING = /^(utm_|fbclid$|gclid$)/i; const clean = (s) => String(s ?? '').trim().replace(/\s+/g, ' '); function normUrl(s) { s = clean(s); if (!s) return ''; const u = new URL(/^https?:\/\//i.test(s) ? s : `https://${s}`); const host = u.hostname.toLowerCase().replace(/^www\./, ''); const params = [...u.searchParams].filter(([k]) => !TRACKING.test(k)).sort(); const qs = params.length ? `?${new URLSearchParams(params)}` : ''; return host + u.pathname.replace(/\/+$/, '') + qs; } const normEmail = (s) => clean(s).toLowerCase(); // Digits only, then the last 9 digits, so "+60 12-345 6789" equals "012-345 6789". const normPhone = (s) => clean(s).replace(/\D/g, '').slice(-9); // The key: the website's domain, or the email's domain if it isn't free mail. function keyOf(r) { const site = normUrl(r.website); if (site) return site.split('/')[0]; const domain = normEmail(r.email).split('@')[1]; if (domain && !FREE_MAIL.has(domain)) return domain; return null; // no reliable key: keep the row as it is } function dedupe(rows, keep) { const byKey = new Map(); const out = []; for (const r of rows) { const k = keyOf(r); if (k === null) { out.push(r); continue; } const prev = byKey.get(k); if (!prev) { byKey.set(k, r); out.push(r); continue; } if (keep === 'newest' && r.scrapedAt > prev.scrapedAt) { out[out.indexOf(prev)] = r; byKey.set(k, r); } } return out; } const rows = JSON.parse(readFileSync(process.argv[2], 'utf8')); console.log('normUrl:', normUrl('http://www.Kopicorner.example/menu/?utm_source=maps&b=2&a=1')); console.log('normPhone:', normPhone('+60 12-345 6789'), normPhone('012-345 6789')); console.log('keys:', rows.map((r) => keyOf(r) ?? '(none)').join(', ')); for (const keep of ['first', 'newest']) { const out = dedupe(rows, keep); console.log(`\nkeep=${keep}: ${rows.length} rows -> ${out.length}`); for (const r of out) console.log(` ${r.scrapedAt} ${clean(r.name).padEnd(15)} ${r.city || '-'}`); }

Run it:

node dedupe.mjs leads.json

Output:

normUrl: kopicorner.example/menu?a=1&b=2 normPhone: 123456789 123456789 keys: acme.example, kopicorner.example, batikhouse.example, acme.example, kopicorner.example, batikhouse.example, nasilemak.example, (none) keep=first: 8 rows -> 5 2026-11-01 Acme Sdn Bhd - 2026-11-01 Kopi Corner Penang 2026-11-01 Batik House Melaka 2026-11-04 Nasi Lemak Co Ipoh 2026-11-04 Roti Studio Ipoh keep=newest: 8 rows -> 5 2026-11-04 ACME Sdn. Bhd. Kuala Lumpur 2026-11-04 Kopi Corner George Town 2026-11-04 Batik House Melaka 2026-11-04 Nasi Lemak Co Ipoh 2026-11-04 Roti Studio Ipoh

The key turned eight rows into five. The first Batik House row has no website, so the script used its email domain, and both Batik House rows got the key batikhouse.example. Roti Studio has only a Gmail address, so it got no key and stayed as a separate row.

Both modes still lose data. "Keep first" lost Kuala Lumpur, the city in Acme's second row. "Keep newest" lost the email address in Batik House's first row. To keep both, merge the duplicates.

3. Merge duplicates into one complete row (Python)

The script below groups rows by the same key, then builds one row per group:

  1. It sorts each group newest first.
  2. It takes each field from the newest copy that has a value, so older copies only fill gaps.
  3. It records a conflict when two copies have different non-empty values for a field, so you can review them.

Two values only conflict if they still differ after normalisation. Without that step, +60 12-345 6789 and 012-345 6789 would conflict. The script uses the phone rule for phone, and case and whitespace rules for most fields. For website it uses a simpler URL rule than dedupe.mjs: it drops the scheme, www., the whole query string and the trailing slash.

Save it as merge.py:

# merge.py - merge duplicate records into one complete row per company. # Python 3.9+. Standard library only. Usage: python3 merge.py leads.json import json, re, sys FREE_MAIL = {"gmail.com", "yahoo.com", "hotmail.com", "outlook.com"} def clean(s): return re.sub(r"\s+", " ", str(s or "")).strip() def key_of(r): site = clean(r.get("website")).lower() if site: site = re.sub(r"^https?://", "", site) return re.sub(r"^www\.", "", site).split("/")[0].split("?")[0] domain = clean(r.get("email")).lower().partition("@")[2] return domain if domain and domain not in FREE_MAIL else None def norm_url(s): s = re.sub(r"^https?://(www\.)?", "", clean(s).lower()) return s.split("?")[0].rstrip("/") NORMALISE = { "website": norm_url, "phone": lambda s: re.sub(r"\D", "", str(s))[-9:], } def same(field, a, b): # Compare with the field's own normaliser; default: ignore case and spaces.  f = NORMALISE.get(field, lambda s: clean(s).lower()) return f(a) == f(b) def merge(group): # Newest copy first, so its non-empty values win. Older copies only fill gaps.  group = sorted(group, key=lambda r: r["scrapedAt"], reverse=True) out, conflicts = {}, set() for r in group: for field, value in r.items(): if not clean(value): out.setdefault(field, value) elif not clean(out.get(field)): out[field] = clean(value) elif field != "scrapedAt" and not same(field, out[field], value): conflicts.add(field) out["_copies"] = len(group) out["_conflicts"] = sorted(conflicts) return out rows = json.load(open(sys.argv[1])) groups, singles = {}, [] for r in rows: k = key_of(r) if k is None: singles.append(r) else: groups.setdefault(k, []).append(r) merged = [merge(g) for g in groups.values()] + singles print(f"{len(rows)} rows -> {len(merged)}") for m in merged: print(json.dumps(m, ensure_ascii=False))

Run it:

python3 merge.py leads.json

Output:

8 rows -> 5 {"name": "ACME Sdn. Bhd.", "website": "acme.example", "email": "sales@acme.example", "phone": "03-1234 5678", "city": "Kuala Lumpur", "scrapedAt": "2026-11-04", "_copies": 2, "_conflicts": ["name"]} {"name": "Kopi Corner", "website": "https://kopicorner.example/menu", "email": "hi@kopicorner.example", "phone": "+60 12-345 6789", "city": "George Town", "scrapedAt": "2026-11-04", "_copies": 2, "_conflicts": ["city"]} {"name": "Batik House", "website": "https://batikhouse.example", "email": "hello@batikhouse.example", "phone": "06-222 3333", "city": "Melaka", "scrapedAt": "2026-11-04", "_copies": 2, "_conflicts": []} {"name": "Nasi Lemak Co", "website": "https://nasilemak.example", "email": "order@gmail.com", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04", "_copies": 1, "_conflicts": []} {"name": "Roti Studio", "website": "", "email": "rotistudio@gmail.com", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04"}

Every company now has one row. No field is empty if any copy had a value, and values that disagree are flagged:

  • Acme takes every value from its newest copy. Its only conflict is name, because ACME Sdn. Bhd. and Acme Sdn Bhd differ by two dots.
  • Kopi Corner has a real conflict: the city was Penang on 1 November and George Town on 4 November. George Town is the capital of Penang state, so both may be correct. Only a person or a better source can decide.
  • Batik House combines the website and phone from 4 November with the email from 1 November.

_copies and _conflicts let you check the result. Sort by _conflicts and review those rows before you trust the merge.

4. Don't use fuzzy matching as your key

Fuzzy matching treats two values as the same when they are similar, not equal. It is tempting for names. Here is Python's built-in difflib scoring four pairs of names. A score of 1.0 means identical.

Save it as fuzzy.py:

# fuzzy.py - why similar names make a risky dedup key. from difflib import SequenceMatcher pairs = [ ("Acme Sdn Bhd", "ACME Sdn. Bhd."), # same company  ("Kopi Corner", "Kopi Korner"), # a typo, or a different shop?  ("Kopi Corner Penang", "Kopi Corner Ipoh"), # two branches  ("Nasi Lemak Co", "Roti Studio"), # clearly different ] for a, b in pairs: score = SequenceMatcher(None, a.lower(), b.lower()).ratio() print(f"{score:.2f} {a!r} vs {b!r}")

Output:

0.92 'Acme Sdn Bhd' vs 'ACME Sdn. Bhd.' 0.91 'Kopi Corner' vs 'Kopi Korner' 0.76 'Kopi Corner Penang' vs 'Kopi Corner Ipoh' 0.25 'Nasi Lemak Co' vs 'Roti Studio'

The same company scored 0.92. A typo, or po...