A release swaps a theme, a plugin or a script bundle, and nobody notices a page got slower until a client says so. Audit the same pages before and after, diff the numbers, and fail the pipeline on a regression. The hard part is defining "worse" when no two Lighthouse runs are identical.

Pick a URL list you can keep

  • Use one page per template: home, a listing, a detail page, an article, any public funnel step.
  • Pin specific items (one product, one post). A "latest posts" page changes between runs.
  • Commit the list. Both runs need the same URLs, device strategy and categories.
  • Keep it public. Google fetches pages from its own servers, so localhost and pages behind a login fail with http-error. Audit production or a public preview URL.

Lighthouse is noisy, so set a tolerance

Two runs of the same page can differ by a few points with no deploy in between. Flag every drop and people soon ignore the check. Three habits keep it honest:

  1. Measure the noise. Audit the unchanged site three times, note the largest swing per metric on your own pages, and set each tolerance above it. The numbers in the script below are placeholders, not recommendations.
  2. Use absolute differences, not percentages: a percentage of a near-zero CLS baseline means nothing.
  3. Confirm before failing. A breach is a candidate: rerun the list once and fail only if the same metric breaches again.

A tolerance hides real changes smaller than itself, so keep each release's result files and glance at the trend.

Gate on lab metrics, check field data later

Lab metrics come from Google's test device, so runs are comparable and they move the moment you deploy. Gate on labMetrics.lcpMs, labMetrics.tbtMs, labMetrics.cls and scores.performance.

Field data is real users from the Chrome UX Report, a 28-day 75th percentile, so a release from ten minutes ago has barely moved it. It cannot gate a deploy. It is null for low-traffic pages, and originFallback: true means origin-level numbers. Use it weeks later: rerun the list and compare fieldData.overallCategory.

Run the audits with the Actor

PageSpeed Insights Audit is an Apify Actor that calls the official PageSpeed Insights API v5 and returns one record per URL and strategy, with output fields documented as stable. Save this as input.json:

{ "urls": [ "https://www.example.com/", "https://www.example.com/products/", "https://www.example.com/blog/a-pinned-post" ], "strategy": "mobile", "categories": ["performance"] }

The diff needs only the performance category. With "strategy": "both" each URL gets a mobile and a desktop record, compared separately. Pages Google cannot audit come back as success: false records with an errorType (dns, timeout, rate-limited and others), unbilled. The script must never read them as "no regression".

The diff script

Save this as diff.mjs (Node only). Exit codes: 0 clean, 1 confirmed regression, 2 some audits could not be compared, 64 bad usage. A metric regresses only if every successful "after" file is past its tolerance, which is how the rerun applies.

import { readFileSync } from "node:fs"; // Placeholder tolerances: replace them with what your own unchanged-site runs show. const checks = [ // label, value, which way is worse (1 = higher, -1 = lower), tolerance ["score", (r) => r.scores?.performance, -1, 5], ["LCP ms", (r) => r.labMetrics?.lcpMs, 1, 500], ["TBT ms", (r) => r.labMetrics?.tbtMs, 1, 200], ["CLS", (r) => r.labMetrics?.cls, 1, 0.05], ]; const [beforeFile, ...afterFiles] = process.argv.slice(2); if (!afterFiles.length) { console.error("usage: node diff.mjs before.json after.json [rerun.json ...]"); process.exit(64); } const load = (file) => new Map(JSON.parse(readFileSync(file, "utf8")).map((r) => [`${r.strategy} ${r.url}`, r])); const before = load(beforeFile); const afters = afterFiles.map(load); if (!before.size) { console.error("the before file has no records"); process.exit(2); } let regressed = 0; let noData = 0; for (const [key, b] of before) { const runs = afters.map((m) => m.get(key)); const good = runs.filter((r) => r?.success); const usable = b.success && good.length && checks.every(([, get]) => [b, ...good].every((r) => Number.isFinite(get(r)))); if (!usable) { noData++; const why = [b, ...runs].map((r) => r?.errorType).filter(Boolean).join(" "); console.log(`NO DATA ${key} ${why || "missing"}`); continue; } const bad = checks.filter(([, get, dir, tol]) => good.every((a) => dir * (get(a) - get(b)) > tol)); for (const [label, get] of bad) console.log(`REGRESSION ${key} ${label}: ${get(b)} -> ${get(good.at(-1))}`); if (bad.length) regressed++; } console.log(`${before.size} audits compared, ${regressed} regressed, ${noData} without data`); process.exit(regressed ? 1 : noData ? 2 : 0);

Run it in CI

A short list fits in one synchronous call. Save this as audit.sh:

set -euo pipefail curl -sSf -X POST \ "https://api.apify.com/v2/acts/josh99smith~pagespeed-insights-audit/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d @input.json -o "$1"

The synchronous endpoint gives up after 300 seconds. The Actor's docs estimate about 10 minutes per 100 URLs, so that covers roughly 50 audits. For longer lists, start the run with POST /v2/acts/josh99smith~pagespeed-insights-audit/runs, poll it, then read its dataset. In GitHub Actions, with the token stored as a secret:

jobs: release: runs-on: ubuntu-latest env: APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }} steps: - uses: actions/checkout@v4 - run: bash audit.sh before.json - run: ./deploy.sh # your own deploy step - run: bash audit.sh after.json - name: Diff, with one rerun to confirm run: | node diff.mjs before.json after.json || { bash audit.sh rerun.json node diff.mjs before.json after.json rerun.json }

If the first diff fails, the job audits again and diffs with both "after" files: a one-off spike passes, a repeat fails. Exit 2 also triggers the rerun, which clears transient quota or timeout failures; if it fails again the build stays red.

Cost and limits

  • Price: $0.004 per successfully audited URL and strategy. Failed pages, invalid URLs and quota errors are free, and each run stops at the maximum cost you set.
  • Worked example: 20 URLs on mobile, before and after, is 40 audits: $0.16. A confirmation rerun adds 20 audits ($0.08), so $0.24 at most. Adding desktop doubles both figures.
  • Quota: the default built-in Google API key, and its daily quota, are shared by all users. When it runs out you get free rate-limited records, which the script reports as exit 2, not a pass. For CI, create your own free key (25,000 audits a day) and pass it as apiKey, built into input.json from a CI secret, not committed.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/pagespeed-insights-audit. PageSpeed Insights and Lighthouse are Google trademarks; the Actor is not affiliated with Google.