In April our checkout started failing for every customer who paid by bank transfer. The cause was a minor version of our payment provider's SDK, merged automatically two days earlier, which had renamed a field in the response we read to get the transfer reference. Our integration suite calls the provider's sandbox and reads exactly that field. It had passed on the pull request. Or rather, it had not failed.
We let Dependabot open pull requests for dependency updates, and a workflow enables auto-merge for patch and minor versions once the required checks are green. The integration tests need a sandbox API key, stored as an Actions secret. Years ago somebody made them skip when the key is missing, with a pytest skipif on the environment variable, so that developers without a key could still run the suite on their laptops.
GitHub runs workflows triggered by Dependabot as if the pull request came from a fork. The GITHUB_TOKEN is read only, and Actions secrets are not passed to the run at all. Dependabot has its own secret store for exactly this reason, and we had never put anything in it. So on every Dependabot pull request the key was empty, all forty two integration tests reported skipped, pytest exited with zero and the check was green. It had been that way for as long as auto-merge had existed, eighteen months. We had been merging every dependency update on the strength of the unit tests alone.
Nobody noticed because a skipped test is drawn in grey, on a page nobody opens when the tick is green.
We pinned the SDK back that morning and had transfers working again within the hour. The sandbox key is now also a Dependabot secret. The skip only applies outside CI: when the CI variable is set and the key is missing, the suite fails at collection with a message saying which secret is absent. The integration job writes the number of tests it ran and the number it skipped to the job summary, and a final step fails the job if more than two were skipped. And auto-merge now requires that job by name, not just a green status on the pull request.
A test that skips itself has decided, on your behalf, that its answer does not matter today. Ours had decided that for every pull request we no longer read.
– Sergey Shinder
