Finding email addresses on a web page is easy. A regex and requests will do it in ten lines. The hard part is everything that regex also finds: the agency that built the site, your@email.com in a form placeholder, a Sentry DSN, a logo@2x.png filename.
When I compared my scraper with the most-used contact scraper on Apify across 63 small-business sites, the difference that mattered most wasn't how many emails we found (57 vs 51 sites). It was that theirs returned 22 addresses belonging to other organisations and mine returned 5. This article is the set of rules that got that number down, with Python you can lift.
1. Start with a strict-enough regex
import re EMAIL_RE = re.compile( r"(?<![\w.+-])" # not glued to another word r"[A-Za-z0-9._%+-]{1,64}" r"@" r"(?:[A-Za-z0-9-]{1,63}\.)+" r"[A-Za-z]{2,24}" r"(?![\w-])" ) Then reject things that match the shape but aren't addresses:
ASSET_EXT = (".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".css", ".js") def plausible(email: str) -> bool: e = email.lower().strip(".") if e.endswith(ASSET_EXT): # logo@2x.png return False local, _, domain = e.partition("@") if len(local) > 40 and re.fullmatch(r"[0-9a-f]+", local): return False # sentry-style hex keys return True 2. Read mailto: links properly
Many addresses only appear in href, sometimes URL-encoded or with a subject attached:
from urllib.parse import unquote, urlsplit def from_mailto(href: str) -> list[str]: if not href.lower().startswith("mailto:"): return [] addr = unquote(urlsplit(href).path) # drops ?subject=... return [a.strip() for a in addr.split(",") if "@" in a] Parse the HTML (BeautifulSoup or lxml) and collect from a[href^=mailto] as well as the visible text. Prefer the mailto version when both exist; the visible text is sometimes a name rather than the address.
3. Undo simple obfuscation
Small businesses write anna [at] example [dot] com to dodge spam bots. Normalise before running the regex:
OBFUSCATIONS = [ (r"\s*[\[\(\{]\s*at\s*[\]\)\}]\s*", "@"), (r"\s*[\[\(\{]\s*dot\s*[\]\)\}]\s*", "."), (r"\s+at\s+(?=[\w-]+\s*(?:\.|\[dot\]|\(dot\)))", "@"), (r"@|@", "@"), ] def deobfuscate(text: str) -> str: for pattern, repl in OBFUSCATIONS: text = re.sub(pattern, repl, text, flags=re.I) return text Be conservative with the bare at rule. "Meet us at the office" shouldn't become an address; the lookahead above only fires when what follows looks like a domain.
4. Decode Cloudflare email protection
Sites behind Cloudflare often have addresses replaced with [email protected] and a link to /cdn-cgi/l/email-protection#..., or a data-cfemail attribute. The encoding is simple: the hex string's first byte is a key, and each following byte is XORed with it.
def cf_decode(hexstr: str) -> str: key = int(hexstr[:2], 16) return "".join( chr(int(hexstr[i:i + 2], 16) ^ key) for i in range(2, len(hexstr), 2) ) # <span class="__cf_email__" data-cfemail="..."> # <a href="/cdn-cgi/l/email-protection#..."> Collect from both the data-cfemail attribute and the fragment after email-protection#. Without this, a good share of sites look like they have no email at all.
5. Drop off-domain addresses (mostly)
This one rule removes most of the junk. The web designer's credit in the footer, the booking platform's support address, the payment provider: none of them are on the site's own domain.
def registrable(domain: str) -> str: # Use tldextract in real code: handles co.uk, com.au etc. import tldextract ext = tldextract.extract(domain) return f"{ext.domain}.{ext.suffix}".lower() def on_site_domain(email: str, site_url: str) -> bool: site = registrable(urlsplit(site_url).hostname or "") return registrable(email.split("@")[1]) == site The exception: small businesses that use Gmail, Outlook or a local ISP address. Keep a short allowlist of free-mail domains (gmail.com, outlook.com, hotmail.com, yahoo.com, gmx.de, web.de, t-online.de and your local ones). Accept those only if they appear in the contact area, header or footer contact block, not inside a blog post or testimonial.
6. Kill template and system addresses
A blocklist for local parts and full addresses you'll see again and again:
JUNK_EXACT = {"your@email.com", "name@example.com", "email@example.com", "you@domain.com", "user@domain.com"} JUNK_DOMAINS = {"example.com", "example.org", "domain.com", "sentry.io", "wixpress.com", "sentry-next.wixpress.com"} JUNK_LOCAL = {"noreply", "no-reply", "donotreply", "mailer-daemon", "example", "test", "postmaster"} def is_junk(email: str) -> bool: e = email.lower() local, domain = e.split("@", 1) return (e in JUNK_EXACT or domain in JUNK_DOMAINS or local in JUNK_LOCAL) Grow these lists from your own output. Every time you see a bad address in results, ask whether it's a one-off or a pattern. Site builders and tracking scripts each have their own; you'll learn them quickly.
7. Know which page section it came from
An address on a team page next to "Founder" means something different from one in a testimonial ("Great service! — tom@clientco.com") or an investors section. When I started recording where on the page each address appeared, I could drop anything from blocks that look like quotes, logos grids or "Our partners", and stop treating testimonial names as staff.
Cheap signals that work: the nearest heading text, CSS classes containing testimonial, review, quote, partner, investor, client, and whether the block sits inside <footer> (contact info, good) or a <blockquote> (bad).
8. Visit the right pages, and few of them
Home, contact, about, team, and the legal notice (/impressum, /imprint, /legal) cover almost everything on small-business sites. Crawling 200 blog pages mostly adds commenters' addresses and guest authors. Limit depth and follow links whose text or path matches those page types, in a few languages.
A note on use
Collecting a published business address is one thing; what you do with it is another. If the output feeds outreach, keep it relevant, low volume, and honour opt-outs. GDPR and PECR apply to personal addresses in the EU and UK, CAN-SPAM in the US.
If you'd rather not maintain it
Those rules, plus founder detection and a firstname@ guess when there's a named owner but no published address, are what I packaged as Founder Email Finder (https://founders.intropi.in, on Apify). Disclosure: I built it. If you're building your own, the eight rules above are where most of the quality comes from.
Written with AI assistance; the numbers come from my own test runs.