To cut node costs, you lowered the requests on your Pods and deployed. Requests are the amount of CPU and memory reserved for each container in a Pod. Allocatable is the amount of a node's CPU and memory that can be assigned to Pods. When you check it yourself, for the resource you lowered, the sum of requests on every node is below half of allocatable.

Cluster Autoscaler (CA) removes a node when the sum of requests on it falls below half of allocatable.1 Yet an hour later the node count hasn't changed, and the CA logs have no error lines (lines starting with E). The bill doesn't go down until nodes are removed.

CA logs, for each node, the reason it decided not to remove it. The line it prints tells you the reason and where to fix it.

I reproduced this on a test cluster, triggered each reason for not removing a node one at a time, and checked which line CA logs for each.2

Check the CA arguments

Before reading the logs, look at --v and --unremovable-node-recheck-timeout in the configuration you installed CA with (Helm values or a manifest).

--v sets how much detail CA writes to its log. Below 4, a node whose sum of requests is at or above half of allocatable in CA's calculation gets no line giving that reason.

--unremovable-node-recheck-timeout (default 5 minutes) is how long CA waits before re-evaluating a node it decided it can't remove because the Pods on it can't be moved to other nodes. During that time, no line appears for that node.3 Set --since in the command in the next section longer than this.

Log lines and reasons

Extract from the CA logs the lines that give the reasons for not removing nodes.

kubectl logs -n <CA namespace> deploy/<CA Deployment name> --since=10m \ | grep -E 'node group min size reached|marked as no scale down|above the scale-down utilization threshold|cannot be removed:|is not suitable for removal:' 

Match each line to the table below by node name. CA checks a node against the table from the first row down and logs only the line for the first reason that matches.4 To evict, in the table, means to move the Pods on a node to other nodes before removing the node. A node group is a set of nodes whose count CA scales up and down; on EKS it corresponds to an Auto Scaling group.

Reason String to grep
The node is not managed by CA None
The node count has reached the node group's minimum size node group min size reached
The node has the scale-down-disabled annotation marked as no scale down
In CA's calculation, the sum of requests is at or above half of allocatable above the scale-down utilization threshold
A Pod can't be evicted cannot be removed:
The Pods have nowhere to go is not suitable for removal:

In the output of the command above, a node that gets no line at all while other nodes do is either not managed by CA or a node CA has decided to remove. A node CA has decided to remove gets a may be removed line, so grep the CA logs for that string as well. If that doesn't appear either, the node is not managed by CA.

How each reason happens

As the previous section says, CA logs only the line for the first reason that matches a node. So after you fix the reason shown, if the same node also matches another reason or another Pod, that line appears next and the node still isn't removed. The top three reasons in the table have nothing to do with requests, so they already applied before you lowered requests.

The node is not managed by CA

A node that isn't in a node group is not evaluated by CA.5 To fix this, let CA find the node group. The AWS example manifest uses --node-group-auto-discovery to look for Auto Scaling groups tagged with k8s.io/cluster-autoscaler/enabled and k8s.io/cluster-autoscaler/<cluster name>, so add these tags to the groups you want CA to manage.6

The node count has reached the node group's minimum size

The line has the form Skipping <node name> - node group min size reached (current: N, min: M), with the current count and the minimum size.5 If you want fewer nodes, lower the minimum size of the Auto Scaling group or its equivalent.

The node has the scale-down-disabled annotation

If a node has cluster-autoscaler.kubernetes.io/scale-down-disabled: "true", CA doesn't remove it.7 If the node is fine to remove, delete this annotation.

In CA's calculation, the sum of requests is at or above half of allocatable

The value you checked can be below half while CA's calculation puts it at or above half. The line shows which resource was at what percentage in CA's calculation.

Node <node name> unremovable: cpu requested (53.75% of allocatable) is above the scale-down utilization threshold 

One cause of the gap is which resource you lowered. CA divides the sum of requests by allocatable separately for CPU and for memory, and checks whether the higher of the two is at or above half.8 If you lowered only memory and CPU is at or above half, CA doesn't remove the node.

The other is sidecars. The requests CA adds up include the requests of sidecars (initContainers with restartPolicy: Always). In kube-state-metrics, kube_pod_container_resource_requests covers only regular containers, and sidecars appear separately in kube_pod_init_container_resource_requests.9 So the following PromQL leaves out the sidecars.

sum by (node) (kube_pod_container_resource_requests{resource="cpu"}) / on (node) kube_node_status_allocatable{resource="cpu"} 

To fix this, review the requests for the resource shown in the log, including the sidecars.

A Pod can't be evicted

The line has the form below. After cannot be removed: come the reason eviction was blocked and the name of the Pod blocking it.10

Node <node name> cannot be removed: <namespace>/<Pod name> is not replicated 

Pods that aren't recreated

If you see is not replicated, the blocker is a Pod that was not created by a ReplicaSet (which a Deployment creates), a StatefulSet, or a Job.11 This covers Pods created directly and Pods created by custom controllers (such as CRD operators). Change them to be created from a Deployment or similar.

Pods involving PDBs

If you see non-pdb-assigned kube-system pod present, the blocker is a kube-system Pod without a PDB. A PDB (PodDisruptionBudget) is an object that sets how many Pods may be stopped at the same time during a drain or a CA eviction. kube-system holds many Pods the whole cluster relies on, such as DNS, so CA doesn't evict a kube-system Pod unless a PDB sets how many of them may be stopped. Since CA 1.33, CA evicts such a Pod without a PDB once the Pod is more than an hour old.12 Add a PDB to that Pod. A Pod with an emptyDir, however, stops at the pod with local storage present line once it has a PDB, and the node is not removed no matter how long you wait. In that case, you also need the fix under "Pods with an annotation or a volume" below.

If you see not enough pod disruption budget to move, the blocker is a Pod whose PDB allows no evictions.13 If kube_poddisruptionbudget_status_pod_disruptions_allowed in kube-state-metrics is 0, that PDB is the cause. If it is 1 or more and this line still appears, CA has decided to remove another node carrying Pods covered by the same PDB, and that node has used up the allowed count. Once that node is removed, the allowed count comes back. A PDB with minAvailable: 1 on a Deployment with replicas 1 keeps the allowed count at 0 at all times. Set replicas to 2 or more and change the PDB to maxUnavailable: 1.

Pods with an annotation or a volume

If you see pod annotated as not safe to evict present, the blocker is a Pod marked so that CA won't evict it.14 This is set with the Pod annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "false", to protect Pods that would cause trouble if stopped partway through their work. If the Pod no longer needs that protection, remove the annotation.

If you see pod with local storage present, the blocker is a Pod that uses an emptyDir (other than medium: Memory) or a hostPath.15 The contents of these volumes are lost when the node is removed, so CA doesn't evict the Pod. If losing the contents is fine, tell CA by putting the volume names in the Pod annotation cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes.

The Pods have nowhere to go

The line has the form below. M is the number of Pods to move off the node, and N is how many were placed before a Pod that doesn't fit came up.

Node <node name> is not suitable for removal: can reschedule only N out of M pods 

CA checks whether the node's Pods fit on the remaining nodes, leaving that node out, using the same conditions as the scheduler.16 Besides free capacity, the conditions include rules on how Pods are placed relative to each other. One of these rules puts only one Pod of the same Deployment on each node, so that losing one node doesn't stop every replica. When the number of replicas equals the number of nodes, whichever node is removed, the remaining nodes have no place for that Pod.

To find the Pod that didn't fit, grep the CA logs for failed to find place for; the Pod name is on that line.17 CA stops trying as soon as one Pod doesn't fit, so this line shows only that one Pod. To fix this, see whether the rule can be changed from "must be followed" to "followed when possible".18

The node group's minimum size, scale-down-disabled, safe-to-evict: "false", and a "must be followed" placement rule can each be a setting someone added to protect something that losing a node would disrupt. Before changing one, I want to ask the person who added it what it was protecting.

What was not checked

This article assumes a CA you installed in the cluster yourself. The managed CA on GKE or AKS does not run inside the cluster, so you can't get its logs with kubectl logs. I did not run a managed CA. I ran only CA 1.36.1 and, to compare how kube-system Pods are handled, 1.32.1, and I used kwok as the cloud provider. I did not run the process that finds AWS Auto Scaling groups by their tags.

  1. The threshold is --scale-down-utilization-threshold, and the default is 0.5 (argument definitions in CA 1.36.1). ↩

  2. On a GitHub Actions ubuntu-24.04 runner, I ran kwok 0.8.0 virtual nodes on top of kind (Kubernetes 1.36.4) and ran CA 1.36.1 with the kwok cloud provider. The CA arguments match the defaults of Helm chart 9.59.0, with only the time arguments shortened. The scripts and results are in show-your-work and the GitHub Actions run. ↩

  3. planner.go and eligibility.go in CA 1.36.1. ↩

  4. The checks in CA 1.36.1 for management and minimum size, the annotation and the sum of requests, and eviction and destinations. ↩

  5. pre_filtering_processor.go in CA 1.36.1. ↩

  6. The AWS example manifest. ↩

  7. eligibility.go in CA 1.36.1. ↩

  8. info.go and pod.go in CA 1.36.1. ↩

  9. kube-state-metrics Pod metrics. ↩

  10. drain.go and rules.go in CA 1.36.1. ↩

  11. The replicated rule in CA 1.36.1. ↩

  12. The system rule in CA 1.36.1. The one-hour check was added in kubernetes/autoscaler#7914. ↩

  13. The pdb rule in CA 1.36.1, and the kube-state-metrics PDB metrics. ↩

  14. The notsafetoevict rule in CA 1.36.1. ↩

  15. The localstorage rule and the volumes it applies to in CA 1.36.1. ↩

  16. cluster.go in CA 1.36.1. ↩

  17. This line appears when --v is 4 or higher. hinting_simulator.go in CA 1.36.1. ↩

  18. The rule goes in the Pod's podAntiAffinity. "Must be followed" is requiredDuringSchedulingIgnoredDuringExecution, and "followed when possible" is preferredDuringSchedulingIgnoredDuringExecution. ↩