Running CDC in Production: The WAL Slot Trap

At 03:02 the primary database ran out of disk and Postgres shut down. There was no runaway query. A CDC connector had been paused nine days earlier, and its replication slot had been holding every byte of write-ahead log since.

This is Part 6 of CDC in Production, the final part. In Part 5 we shipped schema changes without breaking the warehouse. This post is about keeping the pipeline, and the database under it, alive: the WAL slot trap, heartbeats, the three lag numbers that matter, the alerts and runbook you need, and what all of it costs.

Prefer to watch? Here's the 8-minute video version with animated diagrams:

Video link: Running CDC in Production: The WAL Slot Trap

The WAL slot trap

The write-ahead log (WAL) is where Postgres records every change before it counts as committed. A replication slot is a promise about that log. It records the oldest WAL position its consumer still needs, the restart_lsn, and Postgres keeps every WAL file from that point on.

If the connector stops, the slot doesn't. Writes keep coming and the retained WAL keeps growing. And by default there's no ceiling:

# postgresql.conf max_slot_wal_keep_size = -1 # the default: a slot may retain unlimited WAL 

That's the incident. A connector was paused for a migration and nobody resumed it. In this example the database wrote about 40 GB of WAL a day and the disk had 360 GB free, so nine days later it was full. When the disk holding the WAL fills up, the Postgres docs are blunt: the server can panic and shut down.

The quiet slot

There's a sneakier version, with the connector running the whole time.

Debezium confirms its WAL position to Postgres as it commits offsets for the events it sends. Now suppose the captured tables rarely change while the rest of the database is busy. The connector reads the log but sends almost nothing, so the position it confirms falls behind, and Postgres keeps WAL it no longer needs.

The same happens when you capture a quiet database on a server that also hosts a busy one. Slots are per database, but the WAL is shared by the whole server.

On Amazon RDS, even an idle instance generates WAL, because RDS writes to its own system tables about every five minutes.

Heartbeats

The fix is heartbeats. heartbeat.interval.ms defaults to 0, which means off. Set it, say to ten seconds, and the connector emits a heartbeat on that interval, which gives it the chance to confirm the latest position it has read. Postgres can then recycle the old WAL.

For the quiet-database case, go one step further and make each heartbeat a real change in the captured database. Create a tiny table:

CREATE SCHEMA cdc; CREATE TABLE cdc.heartbeat ( id int PRIMARY KEY, beat_at timestamptz NOT NULL ); GRANT USAGE ON SCHEMA cdc TO debezium; GRANT SELECT, INSERT, UPDATE ON cdc.heartbeat TO debezium; -- Without this, the connector never sees the heartbeat rows ALTER PUBLICATION dbz_shop ADD TABLE cdc.heartbeat; 

Then point heartbeat.action.query at it, so the connector runs the statement on every beat:

"heartbeat.interval.ms": "10000", "heartbeat.action.query": "INSERT INTO cdc.heartbeat (id, beat_at) VALUES (1, now()) ON CONFLICT (id) DO UPDATE SET beat_at = EXCLUDED.beat_at" 

The upsert keeps the table at a single row, so it never grows.

Safety valves

Postgres also has safety valves for when everything else fails:

• max_slot_wal_keep_size caps how much WAL a slot may hold. Past that limit the slot can be marked lost, and the connector can't resume from it.
• idle_replication_slot_timeout (Postgres 18 and later) invalidates slots that have sat idle for too long.

You can watch a slot approach the cap in pg_replication_slots.wal_status, which moves from reserved to extended to unreserved and finally lost. The safe_wal_size column tells you how many more bytes can be written before the slot is in danger.

Both valves protect the database at the cost of the pipeline, because a lost slot means a fresh snapshot. That's usually the right trade. Just make it on purpose, before the incident.

Three kinds of lag

You need all three, because each one lies on its own.

1. Slot lag, in bytes. How much WAL each slot is holding:

SELECT slot_name, database, active, wal_status, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_held, pg_size_pretty(safe_wal_size) AS safe_wal_size, inactive_since -- Postgres 17+ FROM pg_replication_slots WHERE slot_type = 'logical' ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC; 

On RDS, the CloudWatch metrics ReplicationSlotDiskUsage and OldestReplicationSlotLag show the same risk.

2. Time lag. Debezium's MilliSecondsBehindSource metric is the gap between a change's commit timestamp and the connector processing it.

3. Sink lag. Consumer lag on the change topics, plus the age of the newest row in the warehouse (freshness).

Why one isn't enough: time lag is measured on events, so when the captured tables go quiet it has nothing new to report, while the slot keeps growing. It also includes any clock difference between the database server and the connector. Slot bytes show the risk to the database. Warehouse freshness shows what your users feel. Put all three on one dashboard.

Alerts, with an owner

Turn those numbers into alerts, and give each alert an owner, a person or rotation that actually gets woken up.

Tune the numbers to your system. The shape is what matters.

The runbook

When the slot alert fires, run the same four steps every time.

1. Look at the slot. Run the query above. Is it active? What's its wal_status? How much WAL is it holding?

2. Check the connector and restart failed tasks:

curl http://connect:8083/connectors/shop-cdc/status curl -X POST \ 'http://connect:8083/connectors/shop-cdc/restart?includeTasks=true&onlyFailed=true' 

If it comes back healthy, watch slot lag fall.

3. If the disk will fill before it catches up, make the call. Drop the slot to save the database, and plan a fresh snapshot. Stop the connector first, because an active slot can't be dropped:

SELECT pg_drop_replication_slot('debezium_shop'); 

When the connector restarts, its stored position is gone. A snapshot mode such as when_needed (from Part 2) takes a new snapshot in that case.

4. Afterwards, reconcile. Run the bucketed reconciliation from Part 3 to prove nothing is missing.

Plan the routine events

Major Postgres upgrades. Slots don't survive a major upgrade, and a new slot only sees changes made after it exists. Debezium's documented procedure, in short:

1. Stop writes to the database.
2. Let the connector drain the slot, then stop it gracefully.
3. Drop the slot.
4. Upgrade.
5. Create the slot again before writes resume, for example with SELECT pg_create_logical_replication_slot('debezium_shop', 'pgoutput');. Otherwise the changes made in between are lost.
6. Check the publication still exists, then restart the connector (the procedure has you recreate it under a new name).
7. Resume writes.

Failover. Since Postgres 17, logical slots can be synchronized to a physical standby. Set slot.failover to true in Debezium, turn on sync_replication_slots (with hot_standby_feedback) on the standby, and list the standby's physical slot in synchronized_standby_slots on the primary. After a promotion, the connector continues on the new primary. Part 3 covers what happens without this.

What it costs

A stuck slot doesn't only risk an outage. On RDS with storage autoscaling, the volume grows to absorb the WAL, and it doesn't shrink back afterwards. Getting allocated storage down later means a blue/green deployment or a new instance.

Elsewhere in the pipeline, Kafka storage grows with retention and replication factor, and in the warehouse every MERGE uses compute. But the cheapest line item is the slot alert, because it protects the database and the bill.

Five common mistakes

1. Leaving a slot behind after pausing or deleting a connector.
2. No heartbeats on a quiet database.
3. Alerting on time lag alone.
4. slot.drop.on.stop=true in production. Debezium's docs say it's for testing only.
5. An alert with no owner, so the page goes to a channel nobody reads at three in the morning.

Recap

• A slot keeps every WAL byte its consumer hasn't confirmed, without limit by default.
• Heartbeats let a quiet connector confirm its position.
• Cap the risk with max_slot_wal_keep_size, and choose that trade on purpose.
• Watch slot bytes, time lag and warehouse freshness together.
• Alert with an owner, and keep a runbook for the slot, upgrades and failover.

This week, run one query: list every replication slot, whether it's active, and how much WAL it holds. If one surprises you, you've found your first incident early.

The CDC in Production series

That's the end of the series. Here's every part, from the basics to this runbook:

• Part 1: Change Data Capture (CDC) Explained (article · video)
• Part 2: Debezium Postgres to Kafka: Every Setting Explained (video)
• Part 3: The Silent CDC Bug: Data Loss With No Errors (video)
• Part 4: CDC Deletes, Tombstones and the Idempotent MERGE (video)
• Previous, Part 5: CDC Schema Changes: What Breaks and How to Ship Them (video)
• Part 6: Running CDC in Production: The WAL Slot Trap (this post · video)

If this was useful, the video version has the animated diagrams for every step: watch it on YouTube. Subscribe to Pipeline Whiteboard for more diagram-first data engineering explainers. Thanks for reading the whole series.