CircleCI outage architecture: PostgreSQL WAL saturation blocked job dispatch

Architecture diagram by BuildOpsy: How PostgreSQL WAL saturation blocked CircleCI's orchestration database, leaving healthy runners idle and thousands of job starts affected.

CircleCI's Runners Were Fine. PostgreSQL Wasn't.

On October 6, 2026, CircleCI hit a failure that had nothing to do with runner capacity.

The culprit was routine PostgreSQL maintenance.

An anti-wraparound autovacuum ran against large tables, generating write-ahead log (WAL) faster than the database could archive it. The WAL volume filled, and the orchestration database went offline for 50 minutes.

The numbers tell the story:

  • ~3,500 job starts failed
  • 50 minutes of database unavailability
  • Hours of recovery work after the database returned

Here's the underlying systems problem.

Let g be the WAL generation rate and a the archival rate.

When g > a, unarchived WAL accumulates. Free disk space only determines how long the system can tolerate that imbalance.

A rough estimate of the remaining time is:

Time to saturation ≈ Free WAL space / (g − a)

This assumes relatively stable rates, but the principle is important: a disk-full alert often arrives too late. By the time it fires, the system may already be out of runway.

The architectural impact was bigger than a database outage.

CircleCI's orchestration database sat on the critical path for workflow state, job dispatch, UI updates, and notifications. The runners could be healthy and ready to execute, but without the control plane coordinating work, they couldn't help.

Then came the second problem: recovery.

Restoring database availability didn't instantly clear the backlog. Queued jobs still needed to be processed and dispatched, putting additional pressure on the system as it recovered.

This is the part of incident response that's easy to underestimate: restoring service and restoring normal operation are two different problems.

The safeguards I'd take from this incident:

  • Monitor WAL generation versus archival throughput. Alert on the rate gap, archive lag, and estimated time to saturation, not just disk utilization.
  • Test autovacuum under realistic workloads. Large tables and production-level write activity can expose bottlenecks that smaller test environments miss.
  • Protect the recovery path. Make sure WAL archival and database recovery have enough capacity and clear operational guardrails.
  • Control backlog redrain. Use bounded concurrency, retry limits, and staggered dispatch to avoid overwhelming downstream services.
  • Design for control-plane failures. A healthy execution fleet is useless if the system responsible for scheduling work is unavailable.

The deeper lesson is that a healthy execution fleet doesn't guarantee a healthy CI/CD system. The control plane that decides what runs can be the real bottleneck.

I broke down the failure mechanism, incident timeline, recovery bottleneck, and architectural trade-offs in the full postmortem:

👉 CircleCI Outage Postmortem: Full WAL Saturation Blocked 3,500 Job Starts

Question for PostgreSQL engineers: Do you monitor WAL generation rate against archival throughput and estimate time-to-full, or are your alerts still mostly based on disk utilization?

I'd be interested to hear what works in production. Thanks