Three checks that look like they passed, but validated nothing

A backup job that dies halfway through a pipe can leave behind a file that reports success, passes its integrity check, and lists its contents correctly — while containing almost none of your data.

I did not want to take that on faith, so I killed a pg_dump on purpose and measured every check I could think of. Three of them passed. Two of those three are the ones people reach for precisely because they are supposed to catch this.

The rig

A throwaway cluster, built for the test and destroyed afterwards, so nothing touched a real database:

initdb -D /tmp/pgtest/data -U test --auth=trust -E UTF8 pg_ctl -D /tmp/pgtest/data -o "-p 55432 -k /tmp/pgtest" start psql -d app -c "CREATE TABLE t(id serial primary key, payload text);" psql -d app -c "INSERT INTO t(payload) SELECT repeat(md5(g::text), 40) FROM generate_series(1,100000) g;" 

That is 100,000 rows of roughly 1.3 KB of text each, dumping in about a quarter of a second. Slow enough to interrupt reliably, fast enough to iterate.

Then the failure, which is the ordinary one — pg_dump dies, gzip keeps going:

pg_dump -d app 2>/dev/null | gzip > app.sql.gz & sleep 0.06 pkill -KILL -f "^pg_dump -d app" 

Check 1: the exit code — passes

pipeline exit status: 0 

A shell pipeline reports the status of its last command. gzip finished cleanly, because the pipe closed and it had nothing left to read. So the pipeline returns 0, and a script written as

pg_dump app | gzip > app.sql.gz 

continues happily to its next line, which is usually "backup succeeded" followed by a rotation step that deletes yesterday's copy.

Check 2: gzip -t — passes

gzip -t app.sql.gz → PASS 

The archive is valid gzip. It is genuinely, correctly compressed. The compression stream has a proper EOF, because gzip was never killed — only its input died. gzip -t verifies the compression, not the payload, and the compression is perfect.

The file was 81,068 bytes against 2,753,525 bytes for the complete dump. 2.9% of the data, zero complaints from the integrity check.

Check 3: pg_restore -l — also passes

This is the one that surprised me. My instinct was that the custom format would be safer, because pg_restore -l reads the table of contents and would obviously notice if anything were missing.

It does not. For -Fc, the TOC is written near the start of the file, before the data blocks. A file truncated during the data phase still has a complete, readable TOC.

full -Fc dump: 2,739,118 bytes, 7 TOC entries truncated dump: 163,840 bytes (6.0%) pg_restore -l truncated.dump → exit 0, 7 entries listed 

Seven for seven. pg_restore -l will tell you an archive contains what it never managed to store.

What actually catches it is trying to use the file rather than inspect it:

pg_restore -f /dev/null truncated.dump → pg_restore: error: could not read from input file: end of file → exit 1 

So the "professional" check and the naive one fail identically. If you swapped gzip -t for pg_restore -l believing you had upgraded your validation, you bought the same blind spot in a better suit.

Aside: pipefail is bash-only, and cron does not use bash

The standard advice is set -o pipefail. It works, and it is also a portability trap:

$ sh -c 'set -o pipefail' sh: 1: set: Illegal option -o pipefail $ ls -l /bin/sh lrwxrwxrwx 1 root root 4 Jan 5 2023 /bin/sh -> dash 

On Debian and Ubuntu, /bin/sh is dash. A script with #!/bin/sh that you tested by hand in an interactive bash session will either die on that line or — worse, if the exit status is not checked — run with the protection silently absent. Cron runs #!/bin/sh scripts under dash, so the environment where the backup actually runs is the one where the guard does not exist.

Either declare #!/usr/bin/env bash, or do not rely on pipelines for correctness at all.

The fix: stop using a pipeline

pg_dump can compress its own output. That removes the entire class of problem, because there is no second process whose exit status can mask the first:

pg_dump -d app -Z6 -f app.sql.gz 

Measured against the same kill:

Two independent signals instead of none, and no pipefail needed.

The reason gzip -t starts working is subtle and useful: when pg_dump compresses internally and is killed, the resulting gzip stream is never finalised — it is missing the CRC32 and ISIZE trailer. In the pipeline version, gzip was never killed, so it wrote a complete, correct trailer around incomplete input. Same truncated payload, opposite integrity verdict.

What actually detects truncation

Ranked by cost, from the measurements above:

The cheapest real guard needs no decompression at all: compare the size against recent history. 81 KB where the last thirty runs were 2.7 MB is not a shape you need to open the file to distrust. A ratio check against a rolling median catches this class of failure for free, and it catches the ones that produce a different exit code from the one you anticipated.

The footer check is the best cheap-and-conclusive option for plain-format dumps, because pg_dump writes -- PostgreSQL database dump complete as its last meaningful line. grep -q for it after decompressing is slower than a size check and far more definitive.

The general shape

This is the same failure as a backup service that exits 0 because it ran, and a monitoring system that is green because nobody defined what red means. The check exists, it runs, it returns success, and it is answering a different question than the one you asked it.

gzip -t answers "is this file valid gzip". pg_restore -l answers "does the header describe some tables". Neither answers "is the data in here".

The only question worth asking is the one that requires touching the data: does the file decompress to a complete dump, and does the restore actually run.

This started as a comment on My Backup Exited 0 for Three Weeks and Nothing Paged Me, where the failure class is the same: a signal that reports the process worked, while the outcome quietly did not. That post covers the freshness side — how a backup can be stale while every timestamp looks recent. This one covers the completeness side.