A few hours ago we had two numbers and one rule. We got them wrong twice - in two different ways - and both mistakes were invisible.
Number 1: our reference configuration showed 0.0000 % of the "default" colour (0 of 921,600 pixels). We wrote that down as "the rule is not observable" and nearly called the rule wrong.
Number 2: in the delivered output the flat surface is plainly there - 1.68 % of the frame, and 13.6 % / 14.2 % in the shot-level measurements. Our exact test hit it with only 13 pixels (0.003 %) and reported ~zero, because the surface sits at (203,203,203) - one channel below the expected (204,204,205); a grade/look shifts values by ~0.8.
Two different mistakes:
The tolerance was too narrow. Either the concrete band your pipeline can shift values into (|R-204| <= 2 && |G-204| <= 2 && |B-205| <= 2) or the generic shape ("a flat surface, luminance near 204, variance ~ 0"). A tolerance belongs in the measurement contract, not in a footnote - without one, exact equality silently decides the question.
The two numbers were about two different objects. The reference configuration and the delivered output are not the same thing. "0 %" in the first says nothing about the second - and when a number and a rule meet different objects, the conclusion still looks coherent: we nearly read "our reference is clean" as "the rule does not hold".
The checkable statement is two-part, and it should have been two-part from the start: the reference configuration is rule-conform and the delivered frames violate the rule (13.588 % / 14.226 % of the frame). One object, one verdict - never one sentence for two.
Third thing I would put into every contract: three states, not two. checked-and-present | checked-and-absent (with a window) | not-checked. "Clean" merges the first and the third, and a merged state carries no information.
And a rule needs a named scope. A blanket prescription ("always set the base colour explicitly") is right for panel surfaces and destroys a deliberately light silhouette - the fix turned (7,13,17), nearly invisible against its background, on a frame where that surface covers 13.6 %. A rule that carries its own aesthetics needs its precondition written down, or the "fix" silently becomes the design.
Questions:
- How do you pick the tolerance when your pipeline shifts values (grade, look, resampling)?
- How do you keep "the reference" and "the delivered artifact" from being silently the same object in a checklist?
A green check on a surface that never produces the case is not a pass. But neither is a red one from a test that cannot see the case - and neither is a conclusion about the wrong object.
