On Tuesday 3 November 2026, at 10:25 UTC, the team behind the Open Build Service started a routine deploy to build.opensuse.org. The Open Build Service, usually shortened to OBS, is the system openSUSE uses to build software packages. A deploy there is not a rare event. This one looked small: a schema migration, a data migration, and some new code.

Six minutes later, the first exceptions landed in the issue tracker.

Some users saw a 500 error on every page they visited. A 500 error is the server saying something went wrong on its side, with no more detail than that. Other users saw nothing wrong at all. The service was up. The deploy had finished. And yet a slice of users could not use the site.

Then came the clue that cracked it. The team removed all the requests from one affected user's watchlist, and that user's errors stopped at once. The problem was not the deploy in general. It was the combination of requests and watchlists. That combination pointed straight at the new code.

What a counter cache is

To understand the bug, you need one piece of machinery: the counter cache.

In OBS, a BsRequest is a request, roughly a proposal to change something, and a BsRequestAction is one action inside that request, such as submitting a package. One request can hold several actions. If you want to know how many actions a request has, you can count them each time you need the number. That works, but it means an extra database query on every page that shows the count.

A counter cache is the shortcut. You add a column to the request that stores the count, and the application keeps that column up to date whenever actions are added or removed. Reading the count becomes reading one number from a row you already have. It is a small optimisation, the kind that appears in thousands of codebases, and it is exactly the kind of change that looks too boring to break anything.

The deploy introduced a counter cache for the number of actions each request has. It did this in three parts: a schema migration that added the count column, a data migration that filled in the real count for every existing request, and new code that read the cached count.

The assumption hiding inside the code

The postmortem states the root cause in two sentences. The cache count defaulted to 0 in the schema migration. The code assumed that every request has at least one action.

Read those two sentences together, because the bug lives in the gap between them.

When the schema migration ran, every existing request got a cached count of 0. Not because those requests had no actions, but because the count had not been calculated yet. Zero was a placeholder, a "not filled in yet" value wearing the clothes of a real number. The data migration that would fill in the true counts started at 10:32, seven minutes after the deploy began, and it had not finished.

Meanwhile the new code was already live, reading the cached count and trusting it. And it trusted something else too: that a request always has at least one action. A request with a cached count of 0 broke that assumption, and the application crashed.

That is why only some users were affected. The crash needed the request-watchlist combination. A user whose watchlist contained requests with a stale zero count hit the new code path and got a 500 on every page. A user whose watchlist held no such requests never touched the code path and saw a perfectly healthy site. The service was not half broken. It was broken for exactly the users who walked into the one place the new code ran.

There is a second detail in the timeline worth pausing on. At 10:46, the team stopped the running data migration, suspecting it was causing the degradation. It was the wrong suspect. The data migration was the fix, not the cause: it was the job that would replace the placeholder zeros with true counts. The root cause was identified at 11:20, almost an hour in. At 11:26 they ran an adjusted data migration by hand. By 11:36 the crashes had stopped, and by 11:45 the corrective migration had finished and the incident was resolved.

So the team lost about half an hour to a reasonable-looking wrong turn. When a migration and a crash arrive together, the migration looks guilty. Here the migration was the only thing that could clear the fault, and stopping it kept the fault in place.

Why zero is the most dangerous number in a deploy

Zero is a value, not an absence. That sentence is the whole lesson, and it is worth saying slowly.

A count of 0 makes a claim: this request has no actions. But during a deploy, 0 can also mean something entirely different: this count has not been calculated yet. The database cannot tell those two meanings apart. Nothing in the number itself says "I am a placeholder." The code that reads it has to know.

This is a pattern that shows up far beyond counter caches. A timestamp of zero can mean "never happened" or "not recorded yet." An empty list can mean "nothing here" or "not loaded yet." A flag set to false can mean "no" or "nobody has answered yet." Every one of these is a value that does double duty as a missing value, and every one of them will eventually be read by code that believes the first meaning.

The OBS postmortem does not publish the exact line of code that crashed, so what follows is the shape of the bug, not a quote of their code. The shape is familiar to anyone who has written it: code that takes the first item from a list without checking whether the list has anything in it.

In Go, the unsafe version looks like this:

first := request.Actions[0] // panics when the list is empty 

The guarded version costs two lines:

// FirstAction returns the first action of a request, or false // when the request has none. Never assume the list is non-empty. func FirstAction(actions []Action) (Action, bool) { if len(actions) == 0 { return Action{}, false } return actions[0], true } 

The same guard in Python:

def first_action(actions): """Return the first action, or None when there is none. A request with no actions is a real state, not an error. Treat it as one. """ if not actions: return None return actions[0] 

Two lines of defence, and neither is clever. That is the point. The fix for this class of bug is not a smarter cache. It is code that treats zero as a value it must handle, because zero is the first value any new counter holds, and the first value any backfill writes.

Deploy in the safe order

The deeper fix is about ordering, not code. The OBS team's own lesson says it plainly: deploying code that immediately depends on a data migration finishing deserves special attention, including planning around a maintenance window and knowing the size of the affected tables.

There are two safe orders, and the deploy used neither.

The first safe order is backfill first, code second. Run the data migration, wait until every row has a true count, and only then ship the code that reads the count. The code never sees a placeholder, because the placeholders are gone before it arrives.

The second safe order is code that tolerates the old state. Ship code that handles a count of 0, or that falls back to counting for real when the cache looks unfilled, and then run the backfill whenever you like. The code never breaks, because it never trusted a number it had not verified.

What broke OBS was the third order: ship the code first, and let the backfill catch up. For a few minutes, or an hour, the code reads numbers that are not true yet. If the code assumes those numbers are true, the deploy is broken from the moment it lands, and the only question is which users walk into it first.

This is the same shape as the expand and contract pattern used for schema changes: first add the new thing without depending on it, then move the data, then remove the old thing. The order is the safeguard. Skip a step and the safeguard is gone.

The test that would have caught it

One test would have caught this before the deploy: a request with no actions.

Test three cases, and test them at the boundary. A request with zero actions, a request with one action, and a request with two. Zero is the case that matters, because zero is the value the schema migration writes for every existing row. If the code cannot handle a request with no actions, it cannot handle a deploy either, because during the deploy every request briefly has no actions, as far as the cache is concerned.

It is a cheap test to write and an easy one to skip, because a request with no actions looks like a case that cannot happen in real life. That is exactly why it happens in production. The cases that cannot happen are the ones nobody guards, and a deploy is very good at making them happen.

What to take away

The deploy was small. A column, a backfill, and some code that read a number. Nothing in it looked dangerous, and nothing in it was exotic. The fault was an assumption so ordinary it was never written down: that every request has at least one action, and that a cached count of 0 means the same thing as a true count of 0.

If you run a counter cache, or any stored number that a migration fills in, the question is not whether your code reads the number. It is whether your code can tell a placeholder from a real value, and whether your deploy order guarantees it never has to. Backfill first, or code that tolerates zero. One of the two, every time.

When did you last deploy code that trusted a number your database had not finished writing?

Sources: Open Build Service team, "Post-mortem: Errors for Users with Requests on their Watchlist", published 13 November 2026, incident 3 November 2026, 10:25 to 11:45 UTC. The postmortem does not publish the exact crashing line of code, so the Go and Python snippets above show the shape of the bug, not OBS source code.