We automated our deploys. Then the phone numbers vanished from the site

We run a website where people book intercity bus tickets. Two weeks ago, shipping a new version meant logging into the server, pulling the code and typing docker compose up -d --build. It worked. It also meant that every release depended on one person being awake and careful.

So we automated it. Fifteen pull requests later, merging a branch puts code on the server by itself.

This is the story of how, and of the five things that broke along the way. The last one is my favourite, because the pipeline did everything right and the site still ended up worse.

What we actually built

In plain words: the branch you merge into decides where the code goes.

Merge into dev → the dev site updates on its own, no questions asked.
Merge into main → a production image gets built and then nothing happens until a human presses a button.

The chain behind that:

merge → GitHub Actions builds an image → image goes to the registry → registry tells the gitops repo → gitops updates the server

The important part is what the server does not do: it never builds anything. Building Next.js on a 3.8 GB machine ran out of memory, so instead of fighting it we moved building to GitHub's runners. The server only downloads a finished image and restarts one container.

Both the dev and the production site live on the same small VPS, in two separate folders, with separate databases. One nginx sits in front. No container is reachable from the internet directly.

Now the five problems.

Problem 1: the safety gate you cannot buy

We wanted production to require approval from a second person. GitHub has that feature. GitHub also said no:

422 Failed to create the environment protection rule. Please ensure the billing plan supports the required reviewers protection rule.

Private repository on a Free plan — no approval rules. And here is the nasty detail: if you create the environment anyway, with no rules in it, it does not pause anything. The job sails straight through. We would have had something that looks like a gate on the screen and is not one in reality.

So the pause became structural instead. Code merged into main triggers no deploy at all. To ship production, someone opens the Actions tab, picks "Deploy", and types the image tag by hand.

The accidental benefit is bigger than the feature we wanted: because you type the tag, production gets exactly the version that was checked on dev, not "whatever happens to be newest when I click".

I will be honest about what this is, though. It is an agreement, not a permission. Anyone with access can deploy to production, and the log records who ran it, never who approved it. A real gate costs a paid plan.

Problem 2: the pipeline that silently did nothing

The application repositories notify the deploy repository with an event. We set it up, merged it to a branch, pushed — and nothing happened. No error. No failed run. No red cross anywhere. The event was accepted and then evaporated.

The rule nobody tells you:

That kind of event always runs the workflow as it exists on the default branch — no matter which branch sent it.

Our deploy workflow was not in main yet. So the event arrived, GitHub looked for a matching workflow on main, found none, and politely did nothing.

This also means the deploy workflow is the one piece of the system you cannot test on a branch first. It only starts working after it is merged. We wrote that in the README in capital letters, because we lost an afternoon to it.

Problem 3: the health check that lies

Our containers have health checks. They were useless.

The check asks the app "are you alive?" from inside the container. For our backend, that endpoint deliberately does not touch the database — otherwise a check running every few seconds would keep the database awake for no reason.

The result: the container happily reports healthy while the database is unreachable. A deploy could finish green with a completely broken site.

So the deploy now knocks on the app from outside, through the real port, and asks a different endpoint — one that does check the database and the queue. It waits up to four minutes, because database migrations run at start-up and the first answer takes a while.

If no answer comes, the deploy puts the previous version back and fails loudly.

One more thing in that order worth copying: download the image before touching the running container. If the tag is a typo, the download fails, the old container keeps serving, and nobody notices. The cheapest rollback is the deploy that never started.

Problem 4: production spent two days pointing at the wrong ports

The configuration files live in git. The server has its own copies. Nothing checks that they match.

They did not match. For two days, the production config in the repository pointed at ports 4000/4001 — which are the dev containers. We found it by reading both files side by side, not because anything alerted us.

This one is still not fixed, and I am not going to pretend otherwise. It is written down in the repo as a known gap. That is the minimum honest position: a problem you have documented gets found during a review; a problem in someone's head gets found by a customer.

Problem 5: the release that removed all the phone numbers

Everything above is infrastructure. This one is the reminder that a green pipeline is not the same as a working site.

After a release, the bus company's phone numbers disappeared from the header and the footer. Not broken, not misformatted — gone. On a website whose entire purpose is getting people onto a bus, with a support line that people actually call.

Here is what visitors saw — reproduced on the live site by making the same API call answer with nothing, since no screenshot survives from the day itself. The little handset icon is still there; the numbers next to it are not:

And here is what it should have looked like, which is also what it looks like now:

What actually happened

Two mistakes stacked on top of each other, and neither was a deploy problem.

First: the database table holding the company's contact details was empty. The seed script that fills it had simply never been run against dev or production. The API did its job correctly and answered {phone1: null, phone2: null, phone3: null}.

Second: the frontend took that answer, turned it into an empty list, and rendered the contact block with nothing inside it. No fallback, no placeholder, no error. An empty answer and a broken answer looked identical to it — like a success.

And a third one lurking underneath, which explains why nobody had noticed the seed was broken: the seed wrote numbers as +38097 480 24 28, with spaces, while the admin panel validates them against a pattern that allows digits only. So even if the seed had run, an administrator could never have saved that row again without retyping every number. The data would have been unusable in a different way.

How it was fixed

The numbers themselves came back by inserting the rows into both databases by hand — production first, dev the next day. That is data, not code, and it needed no release.

Then two pull requests for the real fixes:

The backend seed now writes numbers in the format the validator accepts. Formatting with spaces is a presentation concern and belongs on the frontend, where it already lived.
The frontend now falls back to the known numbers whenever the API gives it nothing at all. The API stays the source of truth — the fallback only covers an empty table or a backend that cannot be reached.

The screenshot above with the numbers in place is the live site right now: nine tel: links, all in the canonical format.

One thing is still open, and it is in the pull request description rather than hidden: the same three numbers are also hardcoded somewhere in the SEO metadata. Fixing that in the same change would have mixed two concerns in one diff, so it waits for its own.

The lesson I actually take from this

Our deploy pipeline worked perfectly throughout. It built the right image, shipped it to the right place, and the health probe came back green — because the app was running, and the API was answering. Correctly, in fact: the table really was empty.

The deploy asks "does the app answer?". It cannot ask "does the page still contain the things a customer needs?". One GET on a health endpoint will never catch a missing phone number.

So the gap is not in the pipeline. It is that an empty answer from an API should never render as an empty page. Degrade to something sensible, and the worst case becomes slightly stale data instead of a support line nobody can see.

The one that has no good answer yet

When a deploy fails, we put the previous image back. But database migrations have already run by then, and they do not reverse.

So a migration that broke the app stays applied, and the older image may not even start against the changed schema. On dev that is annoying. On production it is a rule we wrote down: for releases that contain migrations, do not rely on the automatic rollback. Watch it land on dev first.

What I would take to any project

Let the branch decide the environment. It removes a whole category of "wait, which version is this" conversations.
Download before you touch anything running. A typo in a version tag should cost nothing.
Do not trust a health check that avoids your database. Check from outside, on an endpoint that really looks.
Never render an empty API response as an empty page. This is the cheapest bug to prevent and the most embarrassing one to ship.
Write your known gaps in the repository. The ports mismatch was found by a human reading a list. Nothing automated was ever going to catch it.

About the author

I'm Alex — a DevOps engineer building and shipping the Autolux Bus Station platform: a Next.js storefront, a NestJS API, and the delivery pipeline described above. I write about shipping discipline and the production stories that come with it.