The "distroless" container that had a shell all along
For four months my docker-compose.yml carried a comment like this one:
proof-server: # No healthcheck: the proof-server image is distroless (nix-built) and has # no shell or curl/wget binaries inside, so `docker compose --wait` cannot # gate on it. The deploy script polls the proof-server from the host side # before submitting transactions. image: midnightntwrk/proof-server:8.0.3 Every clause of that comment was wrong except "no curl/wget". The image is not distroless, and it does have a shell: bash, sh, and a full set of coreutils. I had never checked — I had inferred it from the fact that ls doesn't work inside.
That inference made --wait meaningless, and the workaround only covered one deployment path: a polling loop in the deploy script that any other entry point — a second script, a CI job, a teammate's first run — would not inherit.
What the wrong assumption looks like when it runs
Measured on the actual image, same host, same compose file:
| Compose configuration | up --wait returns after | Service ready? |
|---|---|---|
| no healthcheck | 2–4 s | No — /health still refusing connections six seconds later |
| with healthcheck | 13 s | Yes — Up (healthy), /health returns 200 on the next request |
A couple of seconds is the container starting, not the server starting. Any script that trusted --wait and immediately submitted a transaction was racing a process that needs roughly thirteen seconds before it answers.
Ten seconds of checking
The claim was "no shell". The test is one line:
docker run --rm --entrypoint /bin/sh midnightntwrk/proof-server:8.0.3 -c 'echo shell-ok' # shell-ok /bin is not empty either:
docker run --rm --entrypoint /bin/bash midnightntwrk/proof-server:8.0.3 \ -c 'echo /bin/*' # /bin/[ /bin/base64 /bin/bash /bin/cat /bin/cp /bin/date /bin/df ... /bin/sh ... So why did ls fail inside the container and make the image look distroless?
docker run --rm --entrypoint /bin/bash midnightntwrk/proof-server:8.0.3 \ -c 'echo $PATH' # /nix/store/k4a492bl82lg24l8vr6f0mcb77f5hfai-ledger-8.0.3/bin PATH contains exactly one nix store directory — the ledger binaries. Not /bin, not /usr/bin. Typing ls fails with command not found, which is indistinguishable from "there is no ls" unless you look. A minimal PATH is not a minimal image.
What the image genuinely lacks is an HTTP client: no curl, no wget, no nc. That part of the comment was right, and it's the part that matters least — because bash can open a TCP socket by itself.
The healthcheck that works
services: proof-server: image: midnightntwrk/proof-server:8.0.3 environment: PORT: 6300 ports: ["6300:6300"] healthcheck: test: ["CMD", "/bin/bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/6300"] interval: 3s timeout: 3s retries: 20 start_period: 15s /dev/tcp is a bash builtin, not a binary, so it needs no packages and no PATH. The check opens a socket and exits 0 if the kernel connects, non-zero if it doesn't. start_period is generous because the first twelve seconds are the server loading its proving keys — failed checks in that window shouldn't count against the retry budget.
With that in place, docker compose up --wait blocks for about thirteen seconds and returns only once the service is actually listening. The application-side poll stays as a second line of defence for non-compose deployments, but it is no longer the only thing standing between a deploy script and a refused connection.
Two notes if you're adapting this:
- The check is reachability, not correctness. A TCP connect proves the port is bound. It does not prove the server can produce proofs. For that you need a real request — which brings its own trap, because
reading an HTTP response from bash can block on a kept-alive connection and get your healthcheck killed by its timeout. - If you write the HTTP variant in compose, escape
$. Compose interpolates the file before the container sees it, so$linebecomes an empty string and the check silently fails. Write$$line.
When the image really has nothing
Some images genuinely ship no shell: gcr.io/distroless/static, scratch builds, most Go binaries. For those, you have three options, in order of preference:
- Have the application expose health in a way the runtime can see — a
/healthendpoint plus an external checker. The container doesn't need a shell; something outside it does. - Add a healthcheck sidecar — a small image with
curlthat shares the network namespace, run as its own service. - Check from the orchestrator — a Kubernetes
httpGetprobe, or a host-side poll like the one I originally wrote, wired into every deployment path rather than one script.
What you should not do is what I did: assume the image can't be checked, skip the healthcheck, and let --wait mean "the container was created".
The rule
Distroless is a specific claim — no shell, no package manager, only the app and its runtime dependencies. It is checkable in one command, and it is worth checking before you build a workaround around it.
The failure mode here wasn't exotic. It was a PATH with one entry, a command that returned not found, and an assumption that survived four months because nothing forced me to test it. The healthcheck was one line the whole time.
