Reasoning models think in silence. Our job queue read silence as death.

One morning, our daily automated code review flagged a refactor we had just pushed. The finding fit in one line: in the new code, nothing renews a running generation's heartbeat while the model is thinking, so a generation can be reclaimed as dead while it's still working, and the model call is paid for, then thrown away.

It was right, and checking it turned up a hole older than the refactor.

Some context. I run backend engineering at GoodBarber, an app platform, and our AI Extension Builder generates the code of an app section from a description, then edits it on request. Each generation runs as a job on a worker. This post is about how that worker tells the rest of the system it's alive, and why reasoning models broke the way it did.

How a generation says it's alive

A running generation holds a key in Redis that says "I'm alive", with a 90-second expiry. Whoever runs the job is supposed to renew it. The studio, when it polls a generation or when the user reloads the page, checks two things: is the key gone, and is the row older than 180 seconds? If both, the row is reclaimed and marked as failed. That's how we clean up after a worker that crashed mid-job.

The question is what counts as "renewing". In the worker, the key was refreshed when something happened: a token arrived from the model, or the pipeline moved to its next step. Activity meant the job was alive. During a model call that hadn't produced its first token yet, nothing happened, and silence meant it wasn't.

The refactor

We had finished a cleanup the day before and pushed it that morning: every generation would run through the worker, one execution path, and the older paths that ran inside the web request were deleted. About 2,250 lines went away.

One of those paths was the resume after approval. Some generations pause to ask the user for something, a permission or a choice, and resume once the user answers. That resume used to run inside the request, under a small context manager that started a thread renewing the key every 30 seconds, tokens or not. The code around it explained why: if the first token takes more than 90 seconds, the key expires and a live generation gets killed.

The refactor moved the resume onto the worker, which renewed on activity only. The call to the keepalive, and the comment that explained it, were in the code that got deleted.

Why it mattered that week

A reasoning model thinks before it writes. That thinking happens before the first token, and it grows with the reasoning effort you ask for. The day we finished the cleanup, we had also raised the reasoning effort of edits from low to medium and deployed it; edits got about 60% slower. A model that thinks for more than 90 seconds before its first token is, for an activity-driven heartbeat, a model that died.

What happens next depends on who looks first:

• The studio polls, finds the key gone and the row old enough, and marks the generation failed. The worker finishes anyway, and its result is refused. The call is paid for and thrown away, and the user sees an error.
• The stream to the browser decides the worker is dead and starts a second job. The call is paid for twice.

Resumed generations were the most exposed, because their row is created when the plan is made, before the user approves. By the time the job resumes, the row is often already past 180 seconds. The only thing left between it and the reaper is the key.

The older hole

The regression itself never reached production: the review flagged it the morning the refactor was pushed, and the fix shipped with the refactor the next day. But the activity-driven refresh wasn't new. Normal generations on the worker already worked that way. They were less exposed, because a fresh row takes longer to cross 180 seconds. Less exposed is not safe.

We had a failure state for exactly this, a row reclaimed while pending, and it was not empty.

The fix

One change in the worker: every job runs under the same keepalive the old path used, a thread that renews the key every 30 seconds whether tokens arrive or not. The resume, normal generations, edits and retries all go through it. Simplified:

def run_job(generation_id, job): with keep_alive(generation_id, every_seconds=30): return job.run() 

The test makes the worker wait without receiving a single token, and checks that the key was renewed during the wait. Without the fix, it fails with the most honest assertion message I've seen in a while: "0 not greater than 0".

What the numbers say so far

We compared production generations across three periods, read-only, split by deploys: two weeks before the deploy that raised the effort; the two days after it, with the higher reasoning effort already live but neither the refactor nor the fix; and the four and a half days after the next deploy, which shipped the refactor and the fix together. Rates are shares of the generations started by real users in each period, and median times are compared with the first period.

The two later windows last only days, so read their columns as a direction, not a measurement.

Reclaims are what a missing heartbeat produces, whether the worker died or only looked dead, and they nearly disappeared: from 0.6% of generations in the two weeks before to under 0.1% after the fix. The failure rate nearly halved. So far, a fix written for a regression that never shipped also closed the main failure we already had.

One line didn't move, and it belongs in the table anyway. A row left pending is one where the plan was made, nothing was written to the row afterwards, and nobody came back to the page to trigger a reclaim: 0.5% of generations in the two weeks before, 0.6% in the four and a half days after, a rate that didn't improve. Whether its code phase never started or died without a trace, the rows don't say. That isn't the heartbeat. It's the next thing to look at.

The middle column matters too: with the higher effort live for two days, the rate of reclaims didn't go up, 0.3% against 0.6% before. The effort increase alone didn't show up there. It shows up in the last line: at the median, edits took about 60% longer.

Every silence is a budget

The heartbeat is the obvious one. It's not the only thing in a typical stack that reads silence as failure:

• the reaper's age threshold and the key's expiry;
• the stream to the client, if it gives up after a period with no event;
• the reverse proxy: nginx's proxy_read_timeout defaults to 60 seconds;
• the CDN or load balancer in front: Cloudflare returns a 524 when the origin sends nothing for 125 seconds, its current default;
• the worker process timeout, if your app server kills requests that run too long.

When the model started writing within seconds, the silent stretch at the start of a call was short. A reasoning model can stay silent much longer: at medium effort, a first token after 90 seconds was plausible enough for the review to flag it, as a risk rather than a measurement. Each of those timeouts is a budget against thinking time, and raising the reasoning effort spends it.

What I'd keep

• Liveness must not depend on output. A heartbeat comes from a timer. Progress comes from tokens. They are two signals, and the second can stay silent for a long time.
• Raising the reasoning effort is an infrastructure change. List every timeout between the model and the user, and check it against the slowest first token you can expect.
• When you consolidate paths, read what each one was protecting. Our old path carried its guard and the reason for it. The diff that deleted it was green.
• Count "paid and thrown away" as its own failure, not as one more error in the failure rate.

Which timeout in your stack would a model thinking for two minutes hit first?