At 14:00 UTC on October 10, 2026, both of Inngest's database connection poolers lost every connection to the primary database. Not most of them. All of them. And then, at 15:30, it happened again.

Inngest runs background jobs for other companies. When you trigger a function there, a scheduler has to look up your account and your function before any work can start. Those lookups go through the database. With no usable connections, run scheduling stalled and queue processing stopped for most accounts.

The team restarted the poolers. That did not reliably fix it. They raised the pool size. That did not help either. The pool was not too small. It was being emptied faster than it could be refilled, by the very timeouts meant to protect it.

This is a story about a timeout doing exactly what it was told to do, in exactly the wrong place.

The pooler in the middle

First, the pieces, plainly.

A database connection is expensive to open. So most services do not open a fresh one for every request. They keep a pool of open connections and hand them out as needed.

A pooler sits between your service and the database. PgBouncer is one such pooler. Your service connects to PgBouncer, PgBouncer connects to the database, and PgBouncer decides which real database connection serves each query. It lets many application connections share a smaller number of database connections.

A database driver is the library your code uses to talk to the database. Inngest's report points to pgx, the Go driver for Postgres, and to how it interacts with PgBouncer. You do not need to know pgx to follow this story. You only need to know that the driver is the part that sends your query, waits for an answer, and gives up when a deadline passes.

A deadline, in Go, is usually a context with a timeout. You say: this lookup has 200 milliseconds, and if it is not done by then, stop waiting. That sounds safe. It usually is.

The loop that ate the pool

Here is the sequence Inngest describes, slowed down.

Several frequently used lookups had short deadlines. Under normal load, a lookup waits briefly in PgBouncer for a free database connection, gets one, runs, and returns. The deadline never fires.

But suppose the lookup is still waiting in PgBouncer when its deadline expires. The driver cancels the query and closes its connection to PgBouncer. That is the normal, polite thing to do when you have given up.

The problem is that PgBouncer had nothing running to cancel. The query had not started yet. It was still queued, waiting for a connection.

So PgBouncer later ran the abandoned query on a database connection anyway. Then, instead of returning that connection to the pool for reuse, it closed the connection entirely. One cancelled wait, one dead connection.

Now repeat that at scale. Each pooler opens new database connections one at a time. That is a slow refill. If cancellations are destroying connections faster than the pooler can open replacements, the pool drains toward zero. Thousands of clients queue behind the empty pool. Their deadlines expire too. More cancellations. More dead connections. The timeout meant to shed load becomes the thing that keeps the load going.

Inngest reproduced this behaviour in a test environment, so this is not a guess about what might have happened. It is the mechanism the report names.

Two details made October 10 worse. Both collapses started at the top and bottom of the hour, when cron scheduled work roughly triples the query rate. Cron work is work on a schedule, like a job that runs every hour on the hour, and a lot of it lands at once. At the same moments, a new set of lookups that had been rolled out to all accounts before the weekend began failing in large numbers, adding to the cancellations. A spike plus a fresh source of failures, feeding the same loop.

That is also why the obvious fixes failed. Restarting the poolers did not reliably recover them, because the loop restarted with them. Raising the pool size did not help, because opening connections, not the pool limit, was the bottleneck. If you can only refill one connection at a time, a bigger bucket does not help when the bucket has a hole in it.

What actually stopped it

The second collapse ended when Inngest moved all services to connection poolers run alongside the database by its database provider. Processing recovered between 16:10 and 16:25.

That move introduced a new problem, and it is worth naming because it shows how incident fixes trade one risk for another. The new route reached the database through network address translation gateways, and some connections through those gateways were silently dropped. Queries on a dropped connection hung for up to 16 minutes before failing, which intermittently stalled processing, most visibly for cron functions. Inngest moved the affected service back to the original poolers at 04:17 on October 11, and improved the gateways' routing on October 12.

The lasting fix for the pool collapse itself was different, and it is the part you can take home. Inngest changed the lookups involved so that a deadline now ends the caller's wait without cancelling the underlying query. The caller stops waiting. The query is allowed to finish, and its connection goes back to the pool instead of being destroyed. That removes the trigger for this collapse on those lookups. Inngest says it is reviewing the remaining database calls on the execution path and their timeouts, which is the honest version of this fix: it is not done everywhere yet.

The guard you can steal this week

The pattern below is a simplified illustration, not Inngest's code. It shows the split that matters: one deadline for how long the caller waits, and a separate life for the query itself.

The dangerous shape is one deadline doing two jobs:

ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond) defer cancel() // If this is still waiting in the pooler when the deadline fires, // the cancellation can destroy the very connection you were waiting for. row := pool.QueryRow(ctx, "SELECT name FROM accounts WHERE id = $1", accountID)

The safer shape separates the two:

result := make(chan lookupResult, 1) go func() { // queryCtx is deliberately not tied to the caller's deadline. // The query runs to completion and its connection returns to the pool. row := pool.QueryRow(queryCtx, "SELECT name FROM accounts WHERE id = $1", accountID) var name string err := row.Scan(&name) result <- lookupResult{name: name, err: err} }() select { case r := <-result: return r.name, r.err case <-time.After(200 * time.Millisecond): // The caller stops waiting. It does not cancel the query. return "", errors.New("lookup timed out") }

Use this only where it is safe, which is the part people skip. This shape makes sense for a read-only lookup that can safely run to completion even after the caller has given up. It does not make sense as a blanket rule for writes, where a query finishing late could do something the caller already assumed had failed. Check that before you copy it.

Then add three checks around it, because code alone will not save you from the next spike:

  1. Separate your timeouts in your head before you separate them in code. Ask, for each database call: is this deadline for waiting in the queue, for running the query, or for both? If you cannot answer, you have found a candidate for this bug.
  2. Alert on pool health, not just on slow queries. A pool draining toward zero usable connections, and a rising count of cancelled waits, is the early signal. Slow queries alone will mislead you, because in this incident the queries were not slow. They never started.
  3. Load test at the spike, not at the average. Test with the top-of-hour burst, and with a newly deployed lookup failing at the same time. A pool that survives average load can still collapse in the first minute of a burst, and that minute is when everyone is watching.

The wider lesson from the same weekend

October 10 was the second incident in two days, and the first one explains why the second hurt so much. On October 9, from 21:05 to 21:49, one data center provider lost connectivity between its data center and Inngest's other data centers, plus outbound internet access. Inngest runs across three data centers, and function execution depends on all three. Accounts processed in the cut-off data center could not reach the primary database. Accounts elsewhere with run state stored there could not make progress. And accounts with no dependency on that data center were still delayed, because they shared an event-processing pipeline with the affected accounts, and failing work built a backlog there.

Inngest's structural answer is to consolidate onto a single data center that acts as a single cell, which removes this whole class of cross-data-center failure. That is a bigger bet than a timeout fix, and it is still under way. But the two incidents share one lesson: a dependency you treat as background, a shared pipeline or a pooler queue, can become the whole incident when it fails in a way your timeouts and retries were not designed for.

In both cases, the internal failures also counted against functions' configured retries, and a step that failed permanently on an internal error left its run stuck in a Running state instead of marking it failed, so failure handlers were not notified. Inngest has since changed that, so such runs are now marked failed. A stuck state that looks alive is its own kind of outage, quieter than a pool collapse and harder to notice.

So check your own code this week. Find one database lookup with a short deadline that goes through a pooler. Ask what happens if that deadline fires while the query is still waiting, not running. If the answer is "I am not sure", you have found your next test.

What is the shortest deadline in your codebase, and do you know whether it cancels a waiting query or just stops the wait?