Checking one domain is easy: run dig TXT _dmarc.example.com, view the page source, click the padlock. Checking a thousand is a different job. Usually you end up gluing together a DNS script, a tech-detection tool, an SSL checker and a spreadsheet, and then reconciling three output formats.

This guide shows how to do it in one run with one row per domain, using the HireWatch Website Tech Stack Detector + DMARC, SPF, MX & SSL Checker on Apify.

Who this is for

  • Sales and lead-gen teams qualifying accounts by stack: "all the Shopify stores on this list", "uses Shopify + Klaviyo", "on Microsoft 365 behind Proofpoint", "WordPress sites without DMARC".
  • IT, security and email-deliverability folks auditing SPF, DMARC, DKIM and certificate expiry across many domains.
  • Agencies and MSPs running audits across client or prospect domains in one go.
  • SEO and AI-visibility work: robots.txt, sitemap and llms.txt status, and which AI crawlers a site blocks.

What you get per domain

  • Technologies: CMS, ecommerce platform, JS and web frameworks, analytics and tag managers, CDN and hosting, payments, live chat and marketing tools, with versions where the site exposes them. Detection uses Wappalyzer-style fingerprints from the MIT-licensed wappalyzer@6.10.54 set, about 3,588 technologies.
  • Email provider from MX records (Google Workspace, Microsoft 365, Zoho, Proton and others), plus security gateways such as Proofpoint, Mimecast and Cisco, and the mailbox behind them when SPF reveals it.
  • SPF: the record, -all vs ~all, the recursive DNS-lookup count vs the limit of 10, and the sending services.
  • DMARC: policy, subdomain policy, pct and reporting service.
  • DKIM: keys on about 35 common selectors, with key size. MTA-STS, TLS-RPT and BIMI too.
  • DNS: DNS host (Cloudflare, Route 53…), CAA, IPs, and SaaS tools revealed by TXT verification records (Atlassian, DocuSign, Zoom, OpenAI, HubSpot, Stripe…).
  • SSL/TLS: issuer, validity dates, days until expiry, hostname coverage, chain verification, TLS version and key type.
  • robots.txt, sitemap and llms.txt: whether each exists, and which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…) are blocked.
  • HTTP basics: status, redirects, server, response time, HTTP/3, HSTS, security headers, and whether HTTP redirects to HTTPS.
  • Plain-language findings, for example "No DMARC record", "SPF needs 13 DNS lookups (limit 10)", "TLS certificate expires in 9 days" or "No Strict-Transport-Security header on the homepage".

Step 1: Run it in the Apify Console

  1. Open the Actor page and click Try for free (or Start).
  2. Paste your list into domains, one per line. Bare domains (stripe.com), full URLs (https://www.example.com/about) and even email-style inputs (info@example.com) all work, because only the domain is used. Duplicates are removed.
  3. All checks are on by default. You can turn individual ones off (detectTechnologies, checkEmailDns, checkDkim, checkSsl, checkSiteFiles, checkHttpsRedirect), add extra dkimSelectors, or hide weak matches with minTechConfidence (0–100).
  4. Optionally set a Maximum cost per run.
  5. Click Start. Results come in four table views, Overview, Technologies, E-mail & DNS and Audit findings, and you can export them to CSV, Excel or JSON.

Minimal input:

{ "domains": ["apify.com", "stripe.com", "shopify.com"] }

For a 1,000-domain list you can raise maxConcurrency (default 10). Each website still gets at most one request per second.

Step 2: Call it from code

curl 

curl -X POST \ "https://api.apify.com/v2/acts/hirewatch~tech-stack-domain-profiler/runs" \ -H "Authorization: Bearer <YOUR_APIFY_TOKEN>" \ -H "Content-Type: application/json" \ -d '{"domains": ["apify.com", "stripe.com", "shopify.com"]}'

This starts the run asynchronously and returns the run object, including data.id and data.defaultDatasetId. Poll GET /v2/actor-runs/{runId} until status is SUCCEEDED, then download the rows:

curl "https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=csv" \ -H "Authorization: Bearer <YOUR_APIFY_TOKEN>" -o domains.csv

For small lists you can also use the run-sync-get-dataset-items endpoint instead of runs, which waits up to 5 minutes and returns the rows directly. Add ?maxTotalChargeUsd=2 to either URL to cap spend.

Python: find Shopify stores without an enforced DMARC policy

This uses apify-client 3.x (pip install apify-client), where call() returns a Run object. On older 1.x/2.x clients, use run["defaultDatasetId"] instead of run.default_dataset_id.

from apify_client import ApifyClient client = ApifyClient("<YOUR_APIFY_TOKEN>") domains = [line.strip() for line in open("domains.txt") if line.strip()] run = client.actor("hirewatch/tech-stack-domain-profiler").call( run_input={"domains": domains, "maxConcurrency": 20} ) for row in client.dataset(run.default_dataset_id).iterate_items(): if row["status"] not in ("ok", "partial"): continue is_shopify = "Shopify" in (row.get("ecommerce") or "") weak_dmarc = row.get("dmarcPolicy") in (None, "none") if is_shopify and weak_dmarc: print(row["domain"], row.get("emailProvider"), row.get("dmarcPolicy"))

Swap the condition for whatever you're hunting: (row.get("sslDaysLeft") is not None and row["sslDaysLeft"] < 30) for expiring certificates, (row.get("email") or {}).get("spf", {}).get("tooManyLookups") for broken SPF, or "WordPress" in (row.get("cms") or "") for CMS targeting. Each row also has an issues list of {code, severity, message} objects, so any(i["code"] == "DMARC_MISSING" for i in row["issues"]) works too.

What the output looks like

A shortened row from a real run:

{ "domain": "apify.com", "status": "ok", "httpStatus": 200, "hostingCdn": "Amazon Cloudfront, Amazon S3, Amazon Web Services", "emailProvider": "Google Workspace", "spfAll": "-all", "dmarcPolicy": "reject", "dkimFound": true, "sslIssuer": "Amazon", "sslDaysLeft": 97, "hasLlmsTxt": true, "technologyNames": ["Algolia", "Amazon Cloudfront", "HubSpot", "Mixpanel", "React", "..."], "email": { "sendingServices": ["Google Workspace", "Mailgun", "Amazon SES", "HubSpot"], "spf": { "allQualifier": "-all", "dnsLookups": 10, "tooManyLookups": false }, "dmarc": { "policy": "reject", "subdomainPolicy": "reject" }, "dkim": { "found": [{ "selector": "google", "keyBits": 2048 }, { "selector": "pic", "likelySender": "Mailgun", "keyBits": 1024 }] } }, "dns": { "dnsProvider": "Amazon Route 53", "verifiedServices": ["Google Search Console", "OpenAI"] }, "robotsTxt": { "found": true, "aiCrawlersBlocked": [] }, "issues": [] }

Each row gets a status. ok is a full profile. partial means the website was unreachable, returned an error or was disallowed by robots.txt, but DNS, email and SSL data are still returned. domain_not_found and invalid_input are reported but not charged. RUN_REPORT in the key-value store lists the status of every input.

How it works (and its limits)

The profiler requests only the public homepage, /robots.txt, one sitemap, /llms.txt, and a single http:// request to check the HTTPS redirect. It never crawls deeper pages. robots.txt is checked before every request, including each redirect hop. It sends at most one request per second per website, with an honest User-Agent (HireWatchSiteProfiler). DNS and TLS facts come from public DNS and a normal TLS handshake.

No personal data is collected. Page text, email addresses and phone numbers are never output, the mailbox part of DMARC report addresses is masked (***@domain), and no WHOIS data is collected.

Accuracy notes worth keeping in mind:

  • It doesn't run JavaScript in a browser, so tools that only appear after client-side scripts run can be missed. That's the trade-off for speed and cost.
  • DNS can't list DKIM selectors, so only common ones are checked. "Not found" doesn't prove DKIM is missing.
  • Sites behind aggressive bot protection may return 403. You still get DNS, email, SSL and robots data, and the CDN/WAF is usually detected from the response headers.

Pricing

$2 per 1,000 domains ($0.002 per domain profiled). Every check is included, with no add-on charges. Invalid inputs, duplicates and nonexistent domains aren't charged. So a 1,000-domain audit costs about $2, and a 10,000-domain lead list about $20.

Related

If you also care about who's hiring, the HireWatch Workday, Greenhouse, Lever, Ashby & Oracle Jobs Scraper (ATS) pulls open jobs and hiring trends from company career sites, with new and closed jobs flagged between runs. Combined with stack data, it gives you a "who's hiring and what they run" view of an account list.

HireWatch is an independent publisher. This tool is not affiliated with or endorsed by BuiltWith or Wappalyzer. Technology fingerprints come from Wappalyzer (wappalyzer@6.10.54, © Elbert Alias / Wappalyzer), used under the MIT License. Names are used only to describe similar functionality.