Kubernetes Memory Usage and cgroup v2: Telling OOM Kills from Evictions and Right-Sizing Memory Requests

This is an English translation of an article I originally published in Japanese on Qiita.

TL;DR

Scope: Kubernetes v1.37, Linux 6.12, cgroup v2, containerd. Exact versions are listed in Components and versions.

"Memory usage" in Kubernetes is several different numbers, and OOM kills and evictions are each triggered by a different one of them. That is why a container can be OOMKilled while kubectl top shows headroom, or a Pod can be evicted while node utilization looks low.

Terms used throughout (details): memcg OOM is an OOM kill for exceeding a configured limit (container, Pod or /kubepods.slice), and global OOM is an OOM kill caused by memory exhaustion of the whole node. Eviction is a separate mechanism, done by the kubelet.

Key points:

• kubectl top and container_memory_working_set_bytes show the working set, not RSS. It includes active page cache, shmem and kernel memory.
• memcg OOM compares memory.current against the limit, not the working set, and only kills when reclaim cannot bring usage back under the limit.
• Eviction is decided from the node-wide working set, not MemAvailable, so node-exporter utilization can look healthy while the kubelet is close to its threshold.
• OOMKilled does not tell you the path. A SystemOOM event on the Node means global OOM, but its absence proves nothing (events are kept for 1 hour by default).
• The most effective fix for global OOM and eviction is to set the memory request to match actual usage. It does not help with memcg OOM, which needs a limit review.

Where to go next:

• Why do the numbers disagree? Quick reference and Kubernetes memory usage and cgroup v2
• Something was just killed or evicted: Investigating OOM and eviction
• Prevent it: What to do about it, with PromQL for checking requests
• Specific questions: FAQ

Introduction

I get asked about Kubernetes memory usage again and again, and I kept giving the same explanation. The information you need is spread across the Kubernetes and Linux kernel documentation, and I could not find anything that connects the two, so I wrote it down myself.

There is no single definition of "memory usage"

The question "how many MiB of memory is this container using?" has no single correct answer. Unlike CPU utilization or disk usage, memory usage is not one value with one definition. The number changes depending on what you want to measure and where you measure it from.

The reason is that Linux cannot assign every byte of memory to exactly one owner.

• Who owns the page cache? When you read a file, the kernel keeps its contents in memory. The application did not request this with malloc; the kernel allocated it on its own for performance. And when memory runs short, the kernel silently throws it away. Whether it counts as "memory the application is using" depends on your purpose.
• Who owns shared memory? Shared library pages, tmpfs and shared memory are referenced by several processes through the same physical pages. If you simply add up per-process numbers, the same page is counted many times, and the total can even exceed physical memory.
• When does "freed" memory actually go down? Even if an application calls free, the OS-visible usage does not drop unless the allocator or language runtime returns the memory to the OS. If it does return it, usage drops. The value depends on which layer you look at.
• Some memory is reclaimable and some is not. Both count as "in use", but under memory pressure the kernel can take some of it back (page cache) and cannot take back the rest (anonymous pages). You cannot ignore this distinction when estimating free memory.

In other words, memory usage is a value that only becomes defined once you fix the measurement method and the purpose. "The pages a process holds in physical memory", "the memory only that process uses" and "the memory that cannot be freed under pressure" are all legitimate definitions, and each gives a different number.

Kubernetes runs on its own definitions

On top of that, Kubernetes components use their own definitions for management purposes. If you read a value assuming it is the same thing you normally call "memory usage", you will misinterpret it.

The container memory usage you see with kubectl top pod or container_memory_working_set_bytes is not the amount of memory the application has allocated (RSS). Kubernetes treats the working set as container usage, and it includes RSS plus part of the page cache and memory used by the kernel.

container_memory_working_set_bytes # the value kubectl top pod shows ≒ container_memory_rss # memory the application allocated (anonymous pages) + active page cache # only inactive pages are subtracted + tmpfs and shared memory (shmem) # included in full, active or inactive + kernel memory, etc. 

So the working set is larger than what the application actually consumes. The tricky part is that OOM kills and evictions happen because a value other than the one you are looking at reached its limit. For an OOM kill caused by exceeding a limit (called memcg OOM in this article), the kernel compares the container's memory.current against the limit, which is not the working set. And memory.current merely reaching the limit does not cause an OOM; the kill happens when reclaim fails to bring usage back under the limit. Eviction is triggered by free memory computed from the node-wide working set, so it looks at a different scope than the per-container values.

If you treat these as the same thing, you cannot explain situations that look contradictory:

• The working set has not reached the memory limit, yet the container was OOMKilled
• Node memory utilization computed from node_memory_MemAvailable_bytes is low, yet a Pod was evicted
• The application does not leak memory, yet the working set keeps growing
• The RSS the application reports does not match container_memory_rss

None of these are metric bugs. They are the result of each component reporting a value with a definition that suits its own purpose. Conversely, once you know which value has which definition, the apparent contradictions can be explained.

This article covers three things in order:

1. Kubernetes memory usage and cgroup v2: what each value that appears as "memory usage" measures, which cgroup v2 value it corresponds to, and which value triggers eviction
2. How to investigate OOM kills and evictions: there are two OOM kill paths, and how to tell which one happened when you see OOMKilled or an eviction
3. What to do about it: what to do once you know the cause. Right-sizing the memory request is the most effective way to reduce OOM kills and evictions together, so the article focuses on that, including PromQL for checking it

How this article names the two OOM kill paths

There are two paths by which a Pod's process gets killed for memory reasons, and they differ in both cause and blast radius. If you call both "OOM" you cannot tell them apart, so this article uses the following two names throughout.

Put simply, memcg OOM is caused by "your own limit", and global OOM is caused by "the node's free memory". Many cases of OOMKilled despite headroom under the limit are the latter.

The difference in purpose is written in the kernel documentation. Global OOM exists to keep the rest of the system alive; Concepts overview - OOM killer says "In order to save the rest of the system, it invokes the OOM killer". The description of memory.max, on the other hand, is "This is the main mechanism to limit memory usage of a cgroup" (Memory Interface Files), with no mention of protecting other cgroups. cgroup v2 treats limits (memory.max / memory.high) and protection (memory.min / memory.low) as separate models ("implements both limit and protection models"), and memcg OOM belongs to the former.

So memcg OOM can happen even when the node has plenty of free memory. It happens solely because a configured limit was exceeded, regardless of actual pressure. Global OOM, conversely, happens only when node memory is genuinely exhausted, regardless of any limit.

The one exception is the /kubepods.slice hierarchy. Its memory.max is set to Capacity - kube-reserved - system-reserved (see below), which effectively reserves memory for non-Pod processes such as the kubelet and containerd. That is not the purpose of the kernel's memcg OOM; it is Kubernetes using the mechanism that way.

Note: These two names are not official Kubernetes terms. Kubernetes only exposes the termination reason OOMKilled, and it does not distinguish the paths (the reason is explained in Why OOMKilled cannot distinguish the paths).

Both are terms the kernel itself uses, and this article adopts them as they are. The two are used side by side in a kernel comment (mm/oom_kill.c, pagefault_out_of_memory()):

/* * The pagefault handler calls here because some allocation has failed. We have * to take care of the memcg OOM here because this is the only safe context without * any locks held but let the oom killer triggered from the allocation context care * about the global OOM. */ 

The actual distinction is whether the memcg field of struct oom_control is NULL. If it holds a value, the victim is chosen from within that cgroup; if NULL, from the whole node (oom.h). is_memcg_oom() just checks this field.

 /* Memory cgroup in which oom is invoked, or NULL for global oom */ struct mem_cgroup *memcg; 

memcg stands for memory cgroup (memory control group), a short form of struct mem_cgroup in the code above. Variable and function names in the source (is_memcg_oom(), try_charge_memcg()) use the same spelling, and the kernel documentation also writes "Memory cgroup (memcg)" (Documentation/mm/hmm.rst).

Note that the term global OOM appears only in source comments and OOM logs; it is not in the kernel documentation. memcg OOM does appear as "memcg OOM killer" in Documentation/admin-guide/cgroup-v1/memory.rst.

This is how they appear in the kernel log:

Name in this article	How it appears in the kernel log
memcg OOM	constraint=CONSTRAINT_MEMCG, oom_memcg=<cgroup path>, Memory cgroup out of memory: Killed process ...
global OOM	constraint=CONSTRAINT_NONE, ,global_oom, Out of memory: Killed process ...

The prefix of the Killed process line directly indicates the path (oom_kill.c):

 oom_kill_process(oc, !is_memcg_oom(oc) ? "Out of memory" : "Memory cgroup out of memory"); 

Also note that eviction is a separate mechanism from an OOM kill. It is the kubelet, not the kernel, that removes the Pod, and it is observed as Evicted. It is covered below as well.

Quick reference: contradictions explained and a metric map

This section expands the TL;DR: it explains the apparent contradictions from the introduction and maps each metric to the cgroup v2 value behind it.

With that, the seemingly contradictory situations from the introduction can be explained as follows:

• The working set has not reached the limit, yet OOMKilled: memcg OOM compares memory.current, not the working set, against the limit (and merely reaching the limit does not trigger it; the kill happens when reclaim fails). Also, even if your container is under its limit, it can be killed in a node-wide memory exhaustion (global OOM). See My working set is below the limit, but the container was OOMKilled.
• Node utilization computed from MemAvailable is low, yet a Pod was evicted: MemAvailable counts part of the page cache as free, but the kubelet is working-set based and treats all active page cache as "in use". See Node usage computed from MemAvailable is low, but Pods are evicted.
• The application does not leak memory, yet the working set keeps growing: the working set includes active page cache. Page cache is not reclaimed until memory gets tight, so workloads with heavy file I/O or logging look like they keep growing. See Memory working set keeps growing.
• The RSS the application reports does not match container_memory_rss: the former is per-process RSS (including file mappings), and the latter is the anonymous pages charged to the container's cgroup. See The RSS my application reports differs from the RSS metric.

The mapping between these values and the metrics you see in kubectl top and dashboards is as follows. Even among "memory usage" metrics, each one measures something different and corresponds to a different cgroup v2 field.

Looking at the same node, utilization computed from node_memory_MemAvailable_bytes does not match the utilization the kubelet uses for eviction decisions. With workloads that do a lot of file I/O or logging, the difference can reach several GiB.

Different emitters also mean different labels on the metrics. cAdvisor metrics carry no label identifying the node, so when you compute per-node utilization you need to align labels with kubelet_node_name, as shown later.

Finally, the conclusions for diagnosis and remedy:

• To tell which OOM path occurred, check whether the Node has a SystemOOM event. If it does, it is global OOM, but its absence is not evidence of memcg OOM (neither OOMKilled nor metrics distinguish the paths). See Telling which path occurred.
• The most effective remedy is to set the memory request to match actual usage. It affects placement, eviction candidate selection and oom_score_adj at the same time. PromQL for checking is collected in What to do about it.

Kubernetes memory usage and cgroup v2

This section sorts out which cgroup v2 value each metric corresponds to, and which value triggers OOM kills and evictions.

Memory breakdown in cgroup v2

Container memory usage is managed in a per-container cgroup. In cgroup v2, memory.current is the current usage and memory.stat is its breakdown.

# Example values in a container's cgroup (checked on the node) $ cat /sys/fs/cgroup/<container cgroup>/memory.current 5368709120 $ cat /sys/fs/cgroup/<container cgroup>/memory.stat anon 2147483648 # anonymous pages (heap, stack: memory not backed by a file) file 3087007744 # page cache (including tmpfs and shared memory) shmem 268435456 # the part of file that is tmpfs or shared memory ... inactive_anon 134217728 # anon LRU: not accessed for a while active_anon 2013265920 # anon LRU: recently accessed inactive_file 1073741824 # file LRU: not accessed for a while active_file 1744830464 # file LRU: recently accessed unevictable 268435456 # pages that cannot be reclaimed (here, the tmpfs part) ... 

The following relationships hold in this example. Note that inactive_file + active_file does not equal file; it equals file - shmem. The next section explains why.

anon + file + kernel memory = 5368709120 (memory.current) inactive_file + active_file = 2818572288 = file - shmem inactive_anon + active_anon = 2147483648 = anon (shmem is not in here) unevictable = 268435456 = shmem (tmpfs without swap) 

The main fields of memory.stat are:

anon (anonymous pages)

• Memory not backed by a file. Regions the application allocated with malloc and similar belong here.
• Unlike page cache, the kernel cannot reclaim it independently of the process. With swap it can write pages out and free up space, but on nodes that do not use swap it is memory the kernel cannot reclaim under pressure. When swap is unavailable, the kernel sets scan_balance = SCAN_FILE and excludes the anon LRU from scanning (get_scan_count() in mm/vmscan.c).
• On the other hand, this value drops if the process itself returns memory to the OS with munmap or madvise(MADV_DONTNEED). Note that calling free does not reduce it unless the allocator or language runtime returns the memory to the OS.

file (page cache)

• Contents of files held in memory by the kernel during reads and writes. The application did not allocate it explicitly.
• When memory runs short the kernel discards (reclaims) it automatically, so it is basically reclaimable memory.
• Cache of files on disk is split into active_file (recently accessed) and inactive_file (not accessed for a while). Reclaim takes inactive_file first.
• active_file + inactive_file does not equal file. These are totals of the LRU lists, and as described below shmem is on the anon LRU rather than the file LRU. Besides shmem, pages that fall off the LRU, such as mlocked pages, also show up as a difference.
• It includes not only the cache of files on disk but also shmem (tmpfs and shared memory), because Linux treats these as page cache too. shmem is written out to swap rather than to a file, so on nodes without swap it cannot be reclaimed. The breakdown is visible as shmem in memory.stat.

Warning: shmem is counted in file, but it is not on the file LRU. The kernel classifies swap-backed pages as anon, and tmpfs pages are among them. folio_is_file_lru() in mm_inline.h is defined as "0 if @folio is a normal anonymous folio, a tmpfs folio or otherwise ram or swap backed folio".

The counters they are charged to are different, though. A tmpfs page is added to both NR_FILE_PAGES (= file) and NR_SHMEM (= shmem) (mm/shmem.c):

 __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr); __lruvec_stat_mod_folio(folio, NR_SHMEM, nr); 

Furthermore, for a Kubernetes emptyDir with medium: Memory, the kubelet mounts tmpfs with the noswap option (generateTmpfsMountOptions() in empty_dir.go). The kernel makes the inode of a noswap tmpfs unevictable (shmem_get_inode() in mm/shmem.c), so these pages sit on the unevictable LRU, not the anon LRU.

 if (sbinfo->noswap) mapping_set_unevictable(inode->i_mapping); 

So shmem is accounted as follows:

mount	LRU	Where it shows up in memory.stat
emptyDir with medium: Memory (noswap)	unevictable	unevictable
Any other tmpfs / shared memory	anon LRU	inactive_anon / active_anon

In neither case does it enter inactive_file / active_file. That matters in two ways later:

• inactive_file + active_file equals file - shmem, not file
• Only inactive_file is subtracted from the working set, so shmem stays in the working set in full

With workloads that write a lot of logs or read and write large files, the page cache can grow to several GiB even though the application did not intend that.

Note: memory.current includes memory the kernel itself uses (slab, socket buffers and so on) in addition to anon and file. So adding anon and file alone does not give memory.current.

How cgroup v2 memory controls map to Pod settings

cgroup v2 memory control interface

cgroup v2 offers four levels of control over memory usage. memory.min and memory.low are protections ("do not reclaim up to here"), and memory.high and memory.max are limits ("do not allow use beyond here").

Other files that report state:

• memory.current: current usage (what container_memory_usage_bytes is backed by)
• memory.stat: usage breakdown
• memory.events: counts of low, high, max, oom, oom_kill and oom_group_kill
• memory.swap.max: swap limit
• memory.pressure: PSI for waiting on memory allocation

For exact definitions of each file see the kernel's Memory Interface Files, and for how Kubernetes treats cgroup v2 see About cgroup v2.

How Pod settings are reflected

Pod settings are reflected into cgroup v2 as follows.

cgroup v2 is a directory tree. Its top is the root cgroup (/sys/fs/cgroup), where any process that is in no child cgroup belongs, so it contains not only Pods but every process on the node, such as the kubelet and containerd. Kubernetes creates the hierarchy below it: "all Pods on the node → Pod → container".

# With the systemd cgroup driver (the kubeadm default; names differ with the cgroupfs driver, see below) /sys/fs/cgroup ← root cgroup (id="/") │ contains every process on the node │ the value used for eviction decisions │ ├── system.slice/ kubelet, containerd, sshd, etc. │ ├── kubelet.service (processes that are not Pods) │ └── containerd.service │ └── kubepods.slice/ ← all Pods on the node │ (id="/kubepods.slice") ├── kubepods-pod<UID>.slice/ ← Pod (Guaranteed) │ ├── cri-containerd-<ID>.scope ← container │ └── cri-containerd-<pause ID>.scope ← pause container │ ├── kubepods-burstable.slice/ (intermediate slice per QoS class) │ └── kubepods-burstable-pod<UID>.slice/ ← Pod (Burstable) │ └── cri-containerd-<ID>.scope ← container │ └── kubepods-besteffort.slice/ └── kubepods-besteffort-pod<UID>.slice/ ← Pod (BestEffort) └── cri-containerd-<ID>.scope ← container 

When filtering metrics, the root cgroup is id="/", all Pods on the node are id="/kubepods.slice", a Pod is container="", image="", pod!="", and a container is container!="", image!="". What the kubelet uses for eviction decisions is the working set of the topmost root cgroup (see below). That is a different level from memcg OOM caused by a container or Pod exceeding its limit.

Memory limits are set not only on containers but also on higher levels.

A Pod's effective limit is not simply the sum of its regular containers' limits.memory. If there are init containers, their maximum is also considered, and spec.overhead is added if set. For the calculation see Resource sharing within containers and Pod Overhead.

So even if a container has not reached its own limit, an OOM kill can occur when the Pod as a whole or /kubepods.slice hits its limit.

Note: The paths above assume the kubelet uses the systemd cgroup driver. With the cgroupfs driver, paths look like /kubepods/burstable/pod<UID>/<container ID>, with no .slice or .scope and no cri-containerd- prefix on the container.

The kubelet's own default is cgroupfs, but kubeadm sets systemd when the value is empty (defaults.go, kubelet.go). You can check which one is in use with:

$ kubectl get --raw "/api/v1/nodes/<node name>/proxy/configz" | jq '.kubeletconfig.cgroupDriver' "systemd" 

The id label of cAdvisor metrics also changes with the cgroup driver, so when filtering in PromQL, check the value in your own cluster with this query:

group by (id) (container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}) 

A Pod's cgroup path depends on its QoS class. As the tree above shows, Burstable and BestEffort go through an intermediate slice per QoS class.

In the <UID> part, systemd escaping replaces the hyphens of the Pod UID with underscores (example: 6983351e-4ede-... → pod6983351e_4ede_...). The UID of a static Pod (mirror Pod) is a hash with no hyphens, so it looks like pod1ff70b81885169da4e9e32c9530b77c9.slice.

The QoS class and UID of a Pod can be read from the Pod resource, and once you have those two you can assemble the cgroup path.

$ kubectl get pod -n <namespace> <Pod name> -o jsonpath='{.status.qosClass}{"\n"}{.metadata.uid}{"\n"}' Burstable 6983351e-4ede-4611-8bab-b986614af18d # To list them $ kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,QOS:.status.qosClass' 

Container metrics

The cAdvisor built into the kubelet exposes the cgroup values above as Prometheus metrics. They are served from the kubelet's /metrics/cadvisor endpoint, and you do not need to run a separate cAdvisor Pod or DaemonSet.

Note: The kubelet has endpoints such as /metrics, /metrics/cadvisor and /metrics/resource (server.go). The last two relate to container and node usage. /metrics/cadvisor exposes the container_* metrics above, while /metrics/resource exposes only CPU, memory and swap usage (resource_metrics.go). The latter has container_memory_working_set_bytes and also per-Pod and per-node series (pod_memory_working_set_bytes, node_memory_working_set_bytes) (see podMemoryUsageDesc and nodeMemoryUsageDesc in resource_metrics.go). When to use which for per-Pod usage is covered in Align memory request with actual usage.

The important thing here is the definition of the working set.

container_memory_working_set_bytes = container_memory_usage_bytes - container_memory_total_inactive_file_bytes 

In other words, only inactive page cache is subtracted from the working set, and active page cache stays in as "in use". For the implementation see cAdvisor's handler.go.

 workingSet := ret.Memory.Usage if v, ok := s.MemoryStats.Stats[inactiveFileKeyName]; ok { ret.Memory.TotalInactiveFile = v if workingSet < v { workingSet = 0 } else { workingSet -= v } } ret.Memory.WorkingSet = workingSet 

As a breakdown, it is roughly the sum of the following. RSS (container_memory_rss) is exactly anon in memory.stat under cgroup v2, that is, the amount of anonymous pages, so it is included in the working set.

container_memory_working_set_bytes ≒ container_memory_rss (anonymous pages; shmem not included) + shmem (tmpfs and shared memory; stays in full) + container_memory_total_active_file_bytes (active page cache) + unreclaimable pages, kernel memory, etc. (unevictable, slab, socket buffers, ...) 

shmem is listed as a separate term because neither container_memory_rss (= anon = NR_ANON_MAPPED) nor active_file includes shmem. As described above, shmem is not on the file LRU, so it never appears in active_file / inactive_file, and since only inactive_file is subtracted from the working set, it remains in full. No container_* metric is exposed for shmem, so this term can only be checked from memory.stat on the node.

It is ≒ because, besides shmem, there are also no exposed metrics for the kernel memory and locked pages contained in memory.current. For an exact breakdown, read memory.stat on the node directly.

Caution: RSS in container_memory_rss stands for Resident Set Size, but its scope differs from the real meaning. True RSS means "the pages a process holds in physical memory" and includes mmaped file pages.

But memory.stat in cgroup v2 has no rss field, and cAdvisor exposes anon as container_memory_rss. The kernel documentation defines anon as "Amount of memory used in anonymous mappings such as brk(), sbrk(), and mmap(MAP_ANONYMOUS)", which does not include file mappings. So it does not match the RSS column of ps or VmRSS in /proc/[pid]/status.

Warning: A large container_memory_working_set_bytes does not necessarily mean the application needs that much memory. It may just be accumulated page cache. To see how much memory the application has actually allocated, compare against container_memory_rss.

Node metrics

There are two families of metrics representing whole-node memory usage, with different emitters.

From cAdvisor (the kubelet's view)

cAdvisor exposes memory usage not only for containers but also for the root cgroup (id="/", the topmost cgroup that contains every process on the node, as described above) and for the cgroup of all Pods. The id of the all-Pods cgroup depends on the cgroup driver, so the queries below match both values with =~.

• container_memory_working_set_bytes{id="/"}: working set of the whole node (Pods + other processes). But the root cgroup has no memory.current, so the backing value differs from containers (see below)
• container_memory_working_set_bytes{id=~"/kubepods|/kubepods.slice"}: working set of all Pods on the node
• container_memory_total_active_file_bytes{id="/"}: active page cache of the whole node

Both the ...