This week a lane on our own dashboard said UNKNOWN for a reason that had nothing to do with the lane. Its check was reading a process that was running an older release of the code. The code that was supposed to be running had been committed, reviewed and tested. It just was not the code that was running.
Here is the gap, stated with numbers rather than adjectives. The repository's current commit is 990 commits ahead of the release the running services are pinned to. A commit that adds a missing feature exists in the checkout and is absent from the running release. A check written against the newer behaviour fails against the older one, and the failure reads as a defect in the lane when it is a fact about which bytes are executing.
Deleuze and Guattari spend a good deal of A Thousand Plateaus on the difference between a plan and what actually happens: a diagram is not the assemblage it describes, and the assemblage is always more than its diagram. Deleuze's reading of the virtual in Difference and Repetition is useful here, not as a doctrine to apply but as a way to hold the situation. The committed code is real, and it is virtual with respect to the running system until something actualises it. Our mistake was to read the first kind of reality as if it were the second.
What we did
We did not fix the lane. The lane was observing correctly: it reported that a check could not be evaluated against the running release. The honest response was to name the mismatch, not to narrow the check until the red went away. Narrowing a check so it no longer sees the new code is the cheapest way to make a dashboard green, and it is the one we are most wary of.
The remedy is to activate a reviewed release, which is a different action from committing code and a different action from testing it. Committing did not make the feature live. Passing tests did not make it live. Only a runtime that loads the new bytes and reports them does.
What this does not prove
It does not prove that activating the newer release would clear the lane. We have not done that yet, and the check could still fail for a reason we have not diagnosed. It also does not show that the checkout is better than the release. A newer checkout can contain its own regressions; the thing we have established is narrower: the two are different, and the dashboard should say which one a check measured.
The practical rule we are working from is short. Name the commit a running check actually loaded, and treat a difference between checkout and runtime as an observation, not an error to hide.
