We deploy RabbitMQ across single-node and three-node Kubernetes clusters in both staging and production environments to enable asynchronous messaging across our microservices architecture. High availability and predictable message queuing are critical to keeping our downstream processing pipelines operational.


A few weeks back, our infrastructure and SRE teams ran into a frustrating crash loop on one of our high-availability RabbitMQ statefulsets. RabbitMQ liveness probes kept failing randomly and repeatedly, throwing severe thread creation errors into the kubelet logs and causing unexpected pod restarts.


We observed this issue in at least five deployments. The measurements described in this article were collected with RabbitMQ 3.13.3 and Erlang/OTP 26; CPU limits, PID limits, and scheduler counts can differ across deployments.


When inspecting the cluster logs, we repeatedly observed the following log entries emitting from the pod container:

May 21 07:45:39 kubelet[94223]: I1021 07:45:39.876916 94223 prober.go:121] "Probe failed" probeType="Liveness" pod="rabbitmq-0" podUID=c11f6103-8b96-4de4-a649-2a666f957911 containerName="rabbitmq" probeResult=failure output=< May 21 07:45:39 kubelet[94223]: #011Failed to create dirty cpu scheduler thread 138, error = 11 May 21 07:45:39 kubelet[94223]: #011Aborted May 21 07:45:39 kubelet[94223]: #011Failed to create dirty cpu scheduler thread 138, error = 11 May 21 07:45:39 kubelet[94223]: #011Aborted May 21 07:45:39 kubelet[94223]: #011Failed to create dirty cpu scheduler thread 138, error = 11 


Understanding Error 11 (EAGAIN)

To begin troubleshooting, we analyzed the root mechanics of the system call error. In standard Linux error definitions (errno.h), Error 11 maps directly to EAGAIN (Resource temporarily unavailable). When emitted during process creation via fork() or thread creation via pthread_create(), EAGAIN indicates that the OS kernel or execution context refused to allocate a new task because a configured resource limit was reached.


In containerized environments, this error typically points to one of three underlying system constraints:

  • Reaching the OS-level max user process limit (RLIMIT_NPROC).
  • Breaching kernel-level global task limits (sysctl kernel.pid_max).
  • Running into container-level cgroup process limits (pids.max or Kubernetes podPIDsLimit).

Verifying the PID Limit Hypothesis

Our Kubernetes cluster nodes enforce a default pod PID limit of 1024 (podPIDsLimit: 1024). Our initial hypothesis was that the combined workload inside the RabbitMQ pod was breaching this limit and exhausting available task IDs.


To test this hypothesis systematically, we logged into the running container to check standard user limits:

ulimit -u # Output: unlimited 


Because ulimit -u reported unlimited inside the container user namespace, standard POSIX limits were not enforcing the block. However, we

still needed a controlled method to verify whether the container environment was hitting cgroup enforced process cap under execution.


We authored a lightweight bash script (eat_pid.sh) designed to rapidly spawn background sub-processes and hold them open:

  #!/bin/bash if [ -z "$1" ]; then echo "Usage: $0 <number_of_processes>" exit 1 fi NUM_PROCESSES=$1 create_processes() { for ((i=1; i<=NUM_PROCESSES; i++)); do sleep 60 & done } cleanup_processes() { echo "Cleaning up background processes..." pkill -f "sleep 60" } trap cleanup_processes EXIT echo "Creating $NUM_PROCESSES background processes..." create_processes echo "Background processes created. Waiting for 60 seconds..." sleep 60 


We executed ./eat_pid.sh 1000 inside the container, script failed almost immediately with the error: fork: retry: Resource temporarily unavailable. Concurrently, the RabbitMQ runtime threw the exact same log: Failed to create dirty CPU scheduler thread 138, error = 11,


The stress test reproduced EAGAIN when the pod approached its configured PID ceiling. Together with the probe-correlated thread spikes and the improvement after disabling the probes, this supports PID pressure as the likely cause in this environment. A contemporaneous reading of pids.current and pids.max during an incident would provide direct confirmation of the cgroup limit.

Finding the Thread Offender

Having validated that thread exhaustion was occurring, our next step was determining which process inside the container was consuming hundreds of tasks. We opened an interactive session inside the pod and monitored active threads in real time:

top -H -p $(pgrep -f beam.smp)


While observing the task list, we noticed distinct thread spikes occurring every 30 seconds. This 30-second interval matched our RabbitMQ liveness and readiness probe configurations exactly under our initial deployment setup, the health probes executed shell-based rabbitmqctl CLI commands (such as rabbitmqctl status or rabbitmqctl ping) inside the container.


To isolate the probes, we temporarily disabled the liveness and readiness definitions in the pod spec and monitored the baseline thread count. Without the probes, thread allocation flattened into a completely stable baseline. Re-enabling the probes caused immediate thread spikes. The health checks were unambiguously responsible for the task accumulation.

Deep Dive with strace

We needed to profile rabbitmqctl to quantify its exact system call behavior and understand why a basic status check generated massive thread spikes. We used strace to count clone() system calls during a single invocation.


These calls create tasks; the count is useful for comparing runs, but it does not by itself identify every task as an Erlang scheduler thread:

rabbitmq-0$ strace -f -c -q -e trace=clone rabbitmqctl cluster_status % time     seconds  usecs/call     calls    errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00    0.000250          31       700           clone ------ ----------- ----------- --------- --------- ---------------- 100.00    0.000250          31       700           total 


The trace counted more than 700 clone() system calls for one rabbitmqctl invocation in this test. Across the node types we tested, the clone() count increased with the logical CPU capacity visible to the CLI VM. This observation is specific to the tested RabbitMQ/Erlang image and container CPU configuration.


Understanding Erlang OTP Scheduler Mechanics

To understand why a simple CLI utility requires 700 system calls, it helps to examine the internal architecture of the Erlang Runtime System (ERTS), rabbitmqctl is not a self-contained compiled binary (like a Go or C executable). It is an Erlang application wrapper. Launching rabbitmqctl boots an entire Erlang VM instance inside the container. 


By default, the Erlang VM is designed for high-concurrency server workloads and automatically optimizes itself for multi-core hardware. When ERTS initializes, it inspects the operating system to count physical/logical CPU cores and automatically initializes three distinct pools of native OS scheduler threads:

  • Normal Schedulers (+S): Handle standard Erlang process execution, message passing, evaluation, and non-blocking operations. ERTS defaults to spawning 1 normal scheduler per detected logical CPU core.
  • Dirty CPU Schedulers (+SDcpu): Introduced to prevent long-running, CPU-intensive tasks (such as NIFs or heavy C extensions) from stalling normal schedulers. ERTS defaults to spawning 1 dirty CPU scheduler per detected logical CPU core.
  • Dirty I/O Schedulers (+SDio): Designed specifically for synchronous file I/O operations and disk access. The runtime scales this pool dynamically based on core counts and asynchronous thread settings.

The Conflict Between Erlang Auto-Scaling and Kubernetes Containers

This auto-scaling behavior creates a severe architectural friction point inside Kubernetes.


Container runtimes share the underlying host kernel. Unless explicitly restricted at the application runtime layer, Erlang inspects /proc/cpuinfo or host cgroups and detects all CPU cores available on the physical worker node (e.g. 32, 64, or 128 cores), completely ignoring CPU request/limit throttles assigned to the container.


As a result, on a 32-core node, executing rabbitmqctl attempts to initialize 32 Normal Schedulers, 32 Dirty CPU Schedulers, a large Dirty I/O pool, and supporting VM management threads. 


When Kubernetes triggers liveness and readiness probes concurrently every few seconds, multiple rabbitmqctl instances run simultaneously alongside the main RabbitMQ server daemon (beam.smp). Combined, these short-lived VMs spawn hundreds of threads, breaching the pod's 1024 PID cap and causing pthread_create to fail with error = 11.

The Resolution: Overriding CLI Flags via RABBITMQ_CTL_ERL_ARGS

Standard RabbitMQ documentation does not prominently detail CLI thread tuning. However, inspecting the Erlang startup wrapper scripts for rabbitmqctl revealed an environment variable designed for runtime customization: RABBITMQ_CTL_ERL_ARGS.


By defining RABBITMQ_CTL_ERL_ARGS, we can pass explicit Erlang VM startup flags to every rabbitmqctl invocation without altering the thread settings of the primary RabbitMQ server.


We configured the following VM restriction flags:

  • +S 1:1: Sets configured and online normal schedulers to 1.
  •  +SDcpu 1:1: Sets configured and online dirty CPU schedulers to 1.
  •  +SDio 1:1: Restricts dirty I/O schedulers to 1.


We exported the configuration inside the container environment:

 export RABBITMQ_CTL_ERL_ARGS="+S 1:1 +SDcpu 1:1 +SDio 1:1" 

Verification and Results

After exporting the environment variable, we re-ran strace against rabbitmqctl:

rabbitmq-0:/$ export RABBITMQ_CTL_ERL_ARGS="+S 1:1 +SDcpu 1:1 +SDio 1:1" rabbitmq-0:/$ strace -f -c -q -e trace=clone rabbitmqctl cluster_status % time     seconds  usecs/call     calls    errors syscall ------ ----------- ----------- --------- --------- ---------------- 100.00    0.000646          80         8           clone ------ ----------- ----------- --------- --------- ---------------- 100.00    0.000646          80         8           total 

The results were immediate: clone() system calls dropped from ~700 down to just 8 per invocation. Total thread consumption remained negligible, eliminating probe-induced crashes and stabilizing pod execution.

Key Takeaways and Production Best Practices

1: Configure RABBITMQ_CTL_ERL_ARGS in Pod Manifests

Include RABBITMQ_CTL_ERL_ARGS="+S 1:1 +SDcpu 1:1 +SDio 1:1" in your StatefulSet environment definitions if your health checks rely on CLI utilities:

apiVersion: apps/v1 kind: StatefulSet metadata:   name: rabbitmq spec:   template:     spec:       containers:       - name: rabbitmq         image: rabbitmq:3.12-management         env:         - name: RABBITMQ_CTL_ERL_ARGS           value: "+S 1:1 +SDcpu 1:1 +SDio 1:1" 


2. Account for Host-Level Multi-Core Scaling

Be aware that runtimes like Erlang, Java (JVM), and Node.js often auto-detect host physical cores rather than container limits. Always restrict thread pools explicitly in containerized environments.


3. Transition to HTTP Management Probes

Whenever possible, replace CLI exec probes (rabbitmqctl) with native HTTP probes pointing to the RabbitMQ Management Plugin (port 15672). HTTP health checks execute against the existing server process and eliminate subprocess fork overhead entirely. RabbitMQ’s HTTP API requires authentication. Configure probe access using a secure method supported by your deployment, and do not put credentials directly in a plain-text manifest. Verify that these endpoints are available in the RabbitMQ version you run.

livenessProbe:   httpGet:     path: /api/health/checks/virtual-hosts     port: 15672   initialDelaySeconds: 30   periodSeconds: 30 readinessProbe:   httpGet:     path: /api/health/checks/alarms     port: 15672   initialDelaySeconds: 20 periodSeconds: 10