The deploy script already had a rollback. Switch the current symlink, restart the systemd unit, poll /healthz, and if that fails, switch back and restart again. On paper that's a safe deploy.

In practice, every broken release still produced an outage. The reverse proxy returned 502 from the moment the old process stopped until the rollback had brought it back — up to about 35 seconds in the worst case, and a few seconds even after the health check learned to fail fast. A post-switch health check can't prevent that. It runs after the damage, and its only job is to bound it.

The fix was to move the question "does this release start?" to before the switch, and to ask it against the real data.

TL;DR: Before touching current, the deploy now copies the live SQLite database with VACUUM INTO, starts the new release on a free localhost port against that copy with a throwaway config and session secret and every outbound job disabled, and waits for /healthz. If it doesn't come up, the deploy aborts with the test server's last log lines and production never notices. The first check in the preflight is whether the release is readable by the runtime group — because the preflight runs as the deploy user and would otherwise miss permission bugs entirely. A deliberately broken release was then pushed through the real pipeline: the deploy failed, the site served HTTP 200 throughout.

What Does a Post-Switch Health Check Actually Protect?

Less than it looks like. The sequence is:

  1. switch current to the new release
  2. restart the service — the old process is gone now
  3. poll /healthz
  4. on failure: switch back, restart, poll again

Everything between step 2 and the end of step 4 is downtime if the release is broken. The health check decides whether you stay down, which is valuable, but the outage itself is already happening by the time it reports anything.

What breaks a release in a small Node.js app is rarely subtle. A missing package after npm ci, a syntax error in a file the tests didn't load, a schema migration that throws on real data, a file the runtime user can't read. All of those fail within the first second of node server.js. None of them needs the release to be live to show up. They just need the release to start.

How Do You Start a Second Copy Without Touching the First?

Three things have to be separate from production: the port, the data, and anything the app does to the outside world.

The port. Ask the kernel for one instead of picking a number:

port="$(node -e 'const s=require("net").createServer().listen(0,"127.0.0.1",()=>{console.log(s.address().port);s.close()})')"

There's a tiny race between closing that socket and the app binding it, and on a single-tenant VM it doesn't matter. Binding to 127.0.0.1 matters more — the preflight instance must never be reachable from outside, not even for the few seconds it lives.

The data. The app runs on SQLite in WAL mode, which means copying the .db file with cp while the app is running can produce a file that's missing whatever is still sitting in the -wal file, or one that's torn mid-write. VACUUM INTO asks SQLite itself for a consistent snapshot, WAL contents included, as a single self-contained file:

vacuum_into() { TARGET="$1" node -e ' const { DatabaseSync } = require("node:sqlite"); const db = new DatabaseSync(process.env.DB_FILE); db.exec("VACUUM INTO \x27" + process.env.TARGET.replace(/\x27/g, "\x27\x27") + "\x27"); db.close(); ' }

It reads while the live app keeps writing, and it's the same call the deploy already used for its pre-switch backup. Using the real data rather than a fixture is the point: migrations run on startup, and a migration that works on the test schema but fails on a production row is precisely the bug a fixture can't find.

The side effects. This is the part that's easy to get wrong. A second instance of the app is a second instance of everything the app does on startup — including its scheduler. This one syncs orders from an external marketplace every couple of hours and sends mail. A preflight instance that starts its scheduler would make real API calls with real credentials, and if it ever pointed at the wrong database, import orders twice. So the preflight instance is made inert:

( cd "$REL_DIR" && exec env \ PORT="$port" BIND_HOST=127.0.0.1 \ DB_FILE="$tmp/app.db" CONFIG_DIR="$tmp/config" \ NODE_ENV=production EXTERNAL_SYNC_DISABLED=true \ SESSION_SECRET="$(head -c 24 /dev/urandom | od -An -tx1 | tr -d ' \n')" \ node server.js ) >"$tmp/server.log" 2>&1 &

Its config directory holds only the checked-in template, not the local file with SMTP and API credentials. Its session secret is random, so a cookie from the live app means nothing to it. And the sync job has an off switch — which is a design requirement worth adding before you need a preflight, because retrofitting "don't do anything on startup" into an app that assumes it's the only instance is much harder than it sounds.

When Has the Preflight Passed?

When /healthz answers. When the process dies, it has failed — and there's no reason to keep polling a dead PID:

for i in $(seq 1 40); do kill -0 "$pid" 2>/dev/null || break if curl -fsS --max-time 2 "http://127.0.0.1:${port}/healthz" >/dev/null 2>&1; then ok=0; break; fi sleep 0.5 done kill "$pid" 2>/dev/null || true wait "$pid" 2>/dev/null || true

On failure the script prints the last fifteen lines of the test server's log into the CI output, trimmed to 300 characters each. That turned out to be the most useful part of the whole change: a failed deploy now says why in the same place it says that, instead of sending someone to journalctl on the server to find a stack trace from a process that's already been replaced.

/healthz itself is deliberately cheap — it runs SELECT 1 and returns 503 if the database layer throws. By the time it can answer at all, the app has loaded every module, opened the database, and run its startup migrations against the copy. That's the whole set of things this check is for. Whether the business logic is right is the test suite's job, and it ran before the deploy started.

What Can the Preflight Not See?

Being explicit about this kept the check honest.

It runs as the deploy user, not the runtime user. The deploy user owns the release and can read everything in it. The runtime user only shares a group. A file that isn't group-readable boots fine in the preflight and crashes in production — the exact failure from the previous post. So before it starts anything, the preflight checks permissions directly:

bad="$(find "$REL_DIR" \( -type d ! -perm -g=rx -o -type f ! -perm -g=r \) 2>/dev/null | head -n 3)" if [ -n "$bad" ]; then log "files not readable by the group (the app could not open them):" printf ' %s\n' $bad return 1 fi

It doesn't exercise integrations. Outbound sync and mail are off by design. A broken API client starts fine and fails two hours later. That's a trade, not an oversight — the alternative is a preflight with real side effects.

It can't run on the first deploy. No database yet, nothing to copy. The script logs that it's skipping and carries on; the first deploy initialises the schema instead.

It doubles memory for a few seconds. On a small VM that's worth checking once: on a box already near its memory limit, a preflight could trigger the OOM killer on the live process, which would be a fairly ironic way to cause an outage.

How Do You Know It Works?

Two ways, and both are needed.

In the test suite, the deploy script runs for real against a temporary directory layout. npm, sudo and systemctl are stubs; node, curl and bash are real. One release's server.js is a one-line HTTP server that answers /healthz. The other is one line too:

throw new Error('deliberately broken');

The tests assert that the broken release aborts the deploy, that current was never switched, that no pre-switch backup was taken, and that the thrown error text appears in the output. A third case drops group-read from server.js and expects the permission check to name the file; a fourth lets the healthy release through and checks it switches only after preflight. A deploy script with no tests is a deploy script whose failure branches have never run — and the failure branches are the whole reason it exists.

Against the real server, a deliberately broken release went through the real CI pipeline while the site was watched from outside. The deploy went red with the thrown error in its log, and the site answered HTTP 200 the entire time. That's the only result that actually proves the claim, because it's the only one that includes the reverse proxy, systemd, and the real filesystem.

What About the Restart That Still Happens?

A release that passes preflight still needs a restart, and a single-process app on a single VM has a gap of a second or two while that happens. Two small changes make the gap invisible in practice rather than eliminating it:

  • The reverse proxy maps 502/503/504 to a static maintenance page served from outside the release directory, returned as 503 with Retry-After rather than a bare 502. Crawlers and monitors read that as "temporary", humans see a sentence instead of an error page.
  • The frontend retries idempotent GET requests briefly when it sees a 502/503/504, instead of treating any non-2xx as "session expired" and bouncing the user to the login screen — which is what it had been doing.

True zero-downtime would need two processes and a handover, which for an app this size is more machinery than the problem justifies. The preflight moved every predictable failure out of the restart window. What's left is a planned blip, not an outage.

What's the Generalizable Lesson Here?

  • A health check after the switch measures an outage; it doesn't prevent one. Anything that can fail at startup can be checked before the switch, against a non-serving instance.
  • Test against a copy of production data, not a fixture. Startup migrations are the most likely thing to fail on real rows, and the least likely to be covered by test data.
  • Copy a live SQLite database with VACUUM INTO, never cp. In WAL mode a file copy can be missing committed data or be torn mid-write; VACUUM INTO gives you one consistent file while the app keeps running.
  • A preflight instance must be inert: localhost-only, its own port, its own data, a throwaway secret, no schedulers, no outbound calls, no real credentials. Design the off switches in early — an app that assumes it's the only instance is hard to start twice.
  • Write down what the preflight can't see. Here: it runs as the wrong user, so it checks permissions explicitly; it skips integrations, by design.
  • Put the failed instance's last log lines into the CI output. A failed deploy that explains itself in the place it fails saves the trip to the server.
  • Prove it end to end once with a deliberately broken release and an external probe. The test suite proves the script's logic; only the real run proves the proxy, the supervisor and the filesystem agree with it.