A pod that ran for six months crash-looped on its first restart

Our TURN server, the relay that lets two browsers talk when neither can be reached directly, had zero available replicas from about 2026-09-28 until 2026-11-11 11:58 UTC. The container died within a second of starting, every five minutes, about 280 times a day, for around 45 days. Its Deployment had not been touched since March. The alert for it fired on every day Prometheus still has.

This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.

What it looked like

A read-only audit of the cluster (kubegap) that I ran on 2026-11-11 at 05:08 for a different problem listed this one next to it: a Deployment in a production namespace with 0/1 ready and a pod in CrashLoopBackOff with 718 restarts. By 11:40, when I got to it, the number was 794.

$ kubectl -n dam-prod get pod -l app=dam-coturn NAME READY STATUS RESTARTS AGE dam-coturn-549854dfcb-bchx5 0/1 CrashLoopBackOff 794 (2m11s ago) 2d19h dam-coturn-549854dfcb-c8mfp 0/1 ContainerStatusUnknown 28 (2d19h ago) 2d21h dam-coturn-549854dfcb-jgp4h 0/1 ContainerStatusUnknown 11621 (2d21h ago) 235d 

Three pods. The one created in March with 11,621 restarts, a replacement from 2026-11-08 with 28, and the current one with 794. The two older ones had been evicted on 11-08 when the node ran out of disk (that is the previous write-up); the eviction did not cause this, it only produced a new pod that failed the same way.

The previous container's log had the answer in its first line, followed by the program's entire usage text:

$ kubectl -n dam-prod logs dam-coturn-549854dfcb-bchx5 --previous | head -4 turnserver: unrecognized option '--no-dtls' 2026-11-11T11:38:04.215+0000 WARNING Cannot find config file: turnserver.conf. ... 2026-11-11T11:38:04.215+0000 WARNING Cannot find config file: turnserver.conf. ... turnserver: unrecognized option '--no-dtls' 

Started 11:38:04, finished 11:38:04, exit code 255. The binary did not know a flag that was in the Deployment's args, so it printed help and quit. A few hundred lines further down, the same usage text showed what had replaced it:

 --no-tls Do not start TLS client listeners. --dtls Start DTLS client listeners. DTLS is not started by default. 

The alert had been firing since before the retention window. The Alertmanager list that morning had six groups, five of them routed to the Discord receiver and two of those for this namespace:

Which binary was this?

The Deployment said image: coturn/coturn:latest with imagePullPolicy: Always. The pod's status said what that had resolved to:

Image ID: docker.io/coturn/coturn@sha256:bbefd3e1fdfd... 

Docker Hub's tag API maps that digest to 4.18.0, pushed 2026-09-08 11:41, and latest has pointed at it since 11:42, five seconds later. So the running binary was coturn 4.18.0, and the flag it rejected had to have been removed somewhere between whatever version the pod had pulled in March and 4.18.0.

The coturn ChangeLog has one line about DTLS in 4.17.0: "Start the DTLS listeners only when --dtls is given (#2026)". That pull request, merged 2026-08-02, made DTLS opt-in but kept --no-dtls as a deprecated alias that logs a warning. The source at the 4.17.0 tag still has the entry:

/* deprecated: */ {"no-dtls", optional_argument, NULL, NO_DTLS_OPT}, 

At the 4.18.0 tag the line is gone. The commit that removed it is coturn/coturn#2079, merged 2026-09-07 22:43, whose title and ChangeLog entry read "Update help with missing arguments". The body says what it does: "Remove the deprecated --no-dtls and --no-cli turnserver flags — both were already no-ops". Thirteen hours later Docker Hub's latest moved to the image built from it. (The PR says it removed --no-cli too, but its diff only drops that flag from the help text and the test scripts, not from the option table; the 4.18.0 binary still accepts it and logs an ERROR line about it. More on that below.)

When did it start?

Prometheus on this cluster keeps 30 days. The oldest sample for the March pod's restart counter is from 10-12 11:41, and it already reads 4,014. From there the counter climbs by 280 or 281 a day, 284 or 285 on two days, until the eviction on 11-08. Condensed from a one-day-step range query:

kube_pod_container_status_restarts_total{pod="dam-coturn-...-jgp4h"} 10-12 4,014 10-13 4,294 +280 10-14 4,574 +280 ... 11-07 11,314 11-08 11,594 11-09 11,621 evicted 11-08 13:46; counter frozen 

280 a day is one cycle every 308 s (86,400 ÷ 308 = 280.5): the five-minute back-off cap, plus about eight seconds for the kubelet to check the tag with the registry, create the container, watch it exit within a second, and notice. If the slope was the same before the retention window and the counter was near zero before that, 4,014 restarts take 14.3 days, which puts the first failure at about 2026-09-28, three weeks after latest changed. A few restarts could predate it; the date is an estimate. The hourly series in the chart below starts an hour later and reads 4,018; its last point for the current pod is 786, the last scrape before kubectl read 794.

I cannot say why the container restarted on that day after running since 2026-03-20 with, as far as the counter arithmetic shows, no restarts before then. The node has rebooted since (current boot 11-10) and nothing older than 30 days is in Prometheus. One restart was enough: with imagePullPolicy: Always the kubelet asks the registry what the tag points to on every container start, restarts included, and downloads only if the digest changed. The events for the current pod show it, Pulling image "coturn/coturn:latest" 351 times in 29 hours, one per restart. From 09-08 on the answer to that pull was a binary that would not accept the Deployment's arguments.

There was nothing to roll back to, either. The ReplicaSet is the one from March, so no imageID for the March container exists anywhere, and Prometheus does not keep container info that long. Which version had been running for six months is unknown.

The fix

One commit, which Argo CD applied within the minute:

spec: strategy: type: Recreate template: spec: containers: - name: coturn image: coturn/coturn:4.18.0@sha256:bbefd3e1fdfd... imagePullPolicy: IfNotPresent args: # --no-dtls removed  - --no-multicast-peers - --denied-peer-ip=10.0.0.0-10.255.255.255 - --denied-peer-ip=172.16.0.0-172.31.255.255 - --denied-peer-ip=192.168.0.0-192.168.255.255 - --denied-peer-ip=127.0.0.0-127.255.255.255 

The pin is to the digest that was already on the node, so nothing new had to be pulled. The denied-peer-ip ranges were overdue regardless: this relay was coming back onto the internet after six weeks off, and a TURN server that will relay to private addresses, including the relay's own LAN, is a way into the network behind it.

Recreate is there because of the host port. This Deployment uses hostNetwork and declares hostPort: 3478 for UDP and TCP, and the pod is pinned to one node by a nodeSelector. I did not try a RollingUpdate, but the arithmetic says it would not have finished: with one replica, maxUnavailable: 25% rounds down to 0 and maxSurge: 25% rounds up to 1, so the controller has to create the new pod before it may delete the old one; the scheduler will not place a second pod that needs port 3478 on a node where a pod already holds it, crashing or not, and there is no other node it may use. The new pod would sit in Pending and the old one would keep crashing. Recreate deletes first. Every future rollout will have a gap of about 15 s (measured on the rollouts today); with one replica there was never going to be a rollout without one.

$ kubectl -n dam-prod get events --sort-by=.lastTimestamp | grep coturn (condensed) Scaled down replica set dam-coturn-549854dfcb from 1 to 0 Container image "coturn/coturn:4.18.0@sha256:bbefd3e1fdfd..." already present on machine Started container coturn Scaled up replica set dam-coturn-6f99fd45d5 from 0 to 1 $ kubectl -n dam-prod logs deploy/dam-coturn | grep -E "Version|Black|Relay address" INFO Black listing: 10.0.0.0-10.255.255.255 INFO Black listing: 172.16.0.0-172.31.255.255 INFO Black listing: 192.168.0.0-192.168.255.255 INFO Black listing: 127.0.0.0-127.255.255.255 INFO Relay address to use: 192.168.0.101 INFO Coturn Version Coturn-4.18.0 'Gorst' 

A STUN Binding request to port 3478 got no reply at 11:42 and a Binding Success at 11:58:21, both from the LAN and through the router's public address. That checks that the server answers, not that it relays; I did not run an Allocate with the credentials.

Two lines in the new log needed a second commit. ERROR no-cli option is deprecated, see --cli means that flag will go the way --no-dtls went. And WARNING NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED was followed by the server opening 3478 on every interface the host has, including the cluster's and the VPN's. The second commit dropped --no-cli and added --listening-ip for the one address the relay should bind; the next pod's log lists listeners on that address only. A third added the link-local, 100.64/10 and 0/8 ranges to the denied list, since the VPN gives this node an address in 100.64/10. One line, ERROR CONFIG: Unknown argument: with nothing after the colon, is still there and I have not found what produces it.

Fifteen minutes later neither KubePodCrashLooping nor KubeDeploymentReplicasMismatch was active for the namespace, in Prometheus or in Alertmanager:

Timeline

What I am changing

The alert fired for the whole window. KubePodCrashLooping for this Deployment's pods is in every sample Prometheus has. In Prometheus the firing set on 10-12 and on 11-11 was the same list: five CPUThrottlingHigh, KubeCPUOvercommit, KubeDeploymentReplicasMismatch, two KubeJobFailed, KubePodCrashLooping, KubeQuotaFullyUsed, plus KubeMemoryOvercommit from 11-09. Alertmanager inhibits the throttling and quota ones, so what reached Discord was the overcommit alerts, the job failures and this crash loop, re-sent on the route's repeat interval. The crash-loop alert even arrived as a fresh notification three times, once per pod name, and was still not acted on. The previous incident's alert was new and went unnoticed among these standing ones. This one was one of the standing ones.

Done:

• Image pinned by digest, imagePullPolicy: IfNotPresent, strategy: Recreate for the host-network workload.
• --no-dtls and --no-cli gone; --denied-peer-ip for RFC 1918, loopback, link-local, 100.64/10 and 0/8, --no-multicast-peers, --listening-ip.

Not done yet:

• The other 14 images on this cluster referenced by a mutable tag. Each gets the same treatment: digest, IfNotPresent, and a deliberate upgrade step instead of an accidental one.
• The TURN credentials. The relay still authenticates with a static username and password that are compiled into the web client. The standard answer is --use-auth-secret with short-lived credentials minted by the signaling server, which is an application change, not a manifest change.
• The standing alerts, again. Fix or silence each one so that a firing alert means something; page on available == 0 for anything with a public hostname.
• A startup-log check: grep the first lines of a new container's log for ERROR and deprecated and fail the rollout on a match.

For any Deployment on a mutable tag that has been up for months, record what is actually running before anything restarts it: kubectl get pod -o jsonpath='{.status.containerStatuses[*].imageID}'. The tag in the manifest does not tell you.

Public post-incident reports that start the same way, a container restarting until something upstream of it is changed, are collected under the crash loop symptom in Incident Library; the ones where an upgrade removed something the configuration relied on are under breaking change in an upgrade.

Originally published at benchlet.com.