Your visual test failed. Is it the page, or the capture?
A screenshot test fails for two opposite reasons, and the response is opposite:
- The page changed → block the release. That is what the test is for.
- The page did not change, the capture is not reproducible → fix the capture, not the UI. Animation mid-frame, blinking caret, timestamp, ad slot, font fallback, scrollbar.
A gate that cannot separate the two becomes noise, and someone turns it off within a week. Keeping them apart is deterministic by construction: same inputs, same verdict. No AI in the loop.
The four verdicts
| Verdict | Meaning | Action |
|---|---|---|
| PASS | Within the pixel budget and the region limit | Ship |
| FAIL | Budget exceeded, regions listed as x/y/w/h | Look at diff.png and the region crops |
| STALE | The baseline does not belong to this build (git HEAD, mask config or the PNG changed) | Re-baseline deliberately, or unstage the change — never "just re-run" |
| ERROR | Unreadable PNG or size mismatch | That is a capture bug (viewport changed?), not a regression |
STALE exists because the classic failure of visual testing is not flakiness — it is a baseline quietly re-generated until the test went green. Tying the baseline to git HEAD plus a hash of the mask config turns re-baselining into a deliberate, reviewable act.
Exit codes are the contract
0 PASS 1 FAIL 2 STALE 3 ERROR With a --json output you get the verdict, the diff percentage, the region count and every region's bounding box — enough for CI to publish what moved instead of "the test failed".
Budgets that survive contact with reality
-
--max-diffis a safety net, not the primary signal. Antialiasing and font hinting shift a few thousand pixels with zero real change; a real component move shifts tens of thousands. Start at0.001(0.1%). -
--max-regionsis the signal that catches small, meaningful changes. A wrong colour on a 60×40 button is ~0.26% of a 1280×720 page, but exactly one region. Budget:1. - Masks remove whole rectangles and are reported (
masked_pixels) — nobody hides a regression inside one without it showing in the report.
Determinism checklist (before blaming the gate)
- Fixed viewport and DPR 1 — DPR scales the layout and invalidates every baseline
- Disable animation (
prefers-reduced-motion+animation:none!important) - Wait for
document.fonts.readyand network idle — a fallback font diffs the whole page - Hide the caret, the scrollbar and the dev overlay
- Freeze time and randomness
- Mask only what is genuinely dynamic (ad slot, live counter, A/B slot)
- Capture in the same container as CI — a different font stack means different pixels everywhere
The gate has to be tested in both directions
A gate tested only against "nothing changed" is decoration. The one I run has 15 tests, and both sides are asserted — including the case that bites in production: an optimised baseline (optipng, pngquant, ImageMagick) that a naive decoder refuses. The PNG decoder is stdlib (zlib + struct): a CI gate that needs a C library on every machine breaks on the machine that matters.
Full write-up with the exact commands: Your visual test failed: is it the page or the capture?
