you've probably seen a pod go into CrashLoopBackOff after a sudden traffic spike, even though the container's memory request matches the node's capacity. the root cause is an invisible cgroup pressure that the scheduler can't see, and a missing hard limit that lets the process eat the node's RAM until the kernel OOM killer steps in.

the hidden pressure

kubernetes only schedules based on the request field; if you omit a hard limit, the container can exceed its request without any back‑pressure. the node's cgroup will then balloon, starving sibling pods and eventually triggering a node‑wide OOM. the failure surface is hard to diagnose because the pod logs show nothing unusual—only the termination reason "OOMKilled".

apiVersion: v1 kind: Pod metadata: name: memory‑stress spec: containers: - name: app image: myorg/app:latest resources: requests: memory: "256Mi" limits: memory: "512Mi" # explicit hard limit env: - name: ENABLE_METRICS value: "true" - name: cgroup‑exporter image: prom/cgroup-exporter:latest resources: limits: memory: "64Mi" volumeMounts: - name: cgroup mountPath: /host/sys/fs/cgroup volumes: - name: cgroup hostPath: path: /sys/fs/cgroup

the sidecar above scrapes /sys/fs/cgroup and pushes memory usage into Prometheus, giving you a real‑time view of the container's actual consumption versus its limits. coupling this with an alert on container_memory_working_set_bytes approaching container_spec_memory_limit_bytes catches overshoots before the kernel kills the pod.

trade‑offs and failure modes

  • limit too low – the pod throttles under normal load, increasing latency and causing request retries that amplify load. you need to benchmark realistic burst patterns and set the limit a safe headroom (often 1.5‑2× the 95th‑percentile usage).
  • limit too high – you re‑introduce noisy‑neighbor risk; a memory leak in one tenant can starve the rest of the node. run chaos experiments that inject a synthetic leak (stress --vm 2 --vm-bytes 300M) to verify bulkhead isolation.
  • metrics lag – if the exporter polls too infrequently, you may react after the OOM event. configure a scrape interval of ≤15s for critical services.
  • node‑level cgroup fragmentation – on long‑running clusters, leftover cgroup hierarchies can corrupt accounting. schedule a periodic systemd-run --quiet --property=Delegate=yes cleanup job to prune orphaned groups.

by treating the memory limit as a contract rather than a suggestion, you turn an invisible failure domain into an observable one. the contract must be continuously validated in production; otherwise you inherit the same brittle behavior you tried to avoid.

what's your approach to validating pod memory limits in production without triggering OOM kills