This is article 5 of 10 in Beyond Kubernetes Defaults. Read article 4: Kubernetes Secrets Are Distribution, Not Custody first, or start with article 1.

Kubernetes gives us useful workload states: scheduled, Running, Ready, unhealthy, terminating. They are operationally important and precisely defined, but narrower than many application state machines.

The misleading green Pod

Consider an illustrative workload that is Running but unable to participate safely in its application. The process exists, the container started, and a health endpoint answers. Yet participation may still depend on enrollment, current workload identity, correct tenant and placement binding, available credentials, remote admission, desired state, current sessions and durable recovery.

The process is healthy. The workload is not yet admitted.

Five different questions

  • Liveness: Is the process functioning well enough that a restart is not warranted?
  • Kubernetes readiness: Should this Pod be eligible for Service traffic?
  • Application admission: Has the workload joined its system correctly?
  • Application authority: May it perform this operation now?
  • Convergence: Has it applied the state and generation expected of it?

Those answers can legitimately differ. A follower can be healthy and authenticated but not authoritative. An agent disconnected from management can be alive but unable to receive work. An expired credential should fail closed even if its process is reachable.

What the probes actually measure

The Kubernetes probe documentation describes three kinds:

  • A startup probe gates liveness and readiness until it first succeeds.
  • A failing liveness probe triggers a container restart, subject to the Pod's restart policy.
  • A failing readiness probe makes the Pod ineligible for normal Service traffic; it does not restart the container. Exact routing depends on how traffic reaches the Pod, and direct Pod access may behave differently.

None of these alone establishes enrollment, credential validity or the expected application generation. But Kubernetes can express additional state: custom Pod conditions and readiness gates can participate in readiness, and status conditions can carry structured information. API admission controllers and Pod Security Admission answer a different question: whether a Pod specification is permitted, not whether its running application has joined its system.

Define application readiness first. Then decide how probes, conditions and status should reflect it.

One boolean often loses the reason

A generic Ready/not Ready result can hide the cause. Bounded reason codes can distinguish a stale heartbeat, expired credential, missing recovery state, an ahead-of-runtime desired generation, retirement or invalid authorization. Conditions, metrics and status APIs can expose the explanation alongside the traffic decision.

The opposite mistake is making one component's readiness depend on everything downstream. A control component need not become unready because a customer target returns HTTP 500; restarting the control process will not heal that target. A component responsible for delivery may make a different decision. Each layer should derive readiness from the facts it owns.

Startup ordering is not semantic readiness

Init containers completing and dependencies accepting connections do not necessarily mean a workload is ready for its role. An application may still need to issue and verify identity, validate binding, receive desired state, and apply the expected generation. Making those steps explicit avoids sending work merely because a process opened a port.

Make restart behavior testable

A valid ordinary restart should recover the existing identity and state. Re-enrollment is a separate, authorized lifecycle action, not a silent fallback. If recovery state is corrupt or a credential was revoked, a healthy process must not silently regain authority.

Useful restart assertions include identity continuity, no generation rollback, no resurrection of retired state, no leader-only action by a follower, and a bounded reason while convergence is incomplete.

An illustrative scenario

A queue worker starts, returns 200 from its health endpoint and passes readiness. The Service sends an event, but the worker discovers its session credential expired during startup. If the only signal is a green Pod, the instinct may be to restart it. With explicit application state, the worker can report the expired credential and leave its traffic-eligible state; the operator sees that re-issuance, not a restart, is needed.

Observation and traffic eligibility can remain separate: a health endpoint can expose process status, readiness can govern Service traffic, and richer admission and convergence state can be available through status or metrics.

A practical checklist

  • Does liveness answer process questions?
  • Does readiness derive from facts this component owns?
  • Are admission and convergence observable with useful reasons?
  • Is restart recovery distinguishable from re-enrollment?
  • Does a downstream failure affect only the layers that own it?
  • Can an operator distinguish a stopped process from a healthy follower or an unadmitted workload?

Running and Ready have precise meanings. Define admission and authority for your application, expose them as first-class state, and let Kubernetes probes reflect that model.