The cloud bill for your Kubernetes cluster went up again. You add nodes because new pods sit in Pending with "Insufficient cpu". Then you open a dashboard and the nodes are using 20 percent of their CPU.

Both things are true at the same time, and the reason is simple. The scheduler doesn't place pods based on what they use. It places them based on what they request. If every team asks for four times what their service needs, the cluster fills up on paper while the hardware sits mostly idle, and you pay for every one of those empty cores.

This article walks through finding that gap, closing it safely, and cleaning up the idle capacity that builds up around it.

Why requests, not usage, drive your bill

Every container can declare resource requests:

resources: requests: cpu: "1" memory: 2Gi

The scheduler treats those numbers as reserved. A node with 4 CPUs can hold four pods that each request 1 CPU, even if each of them actually uses 0.1. A fifth pod goes Pending, the cluster autoscaler adds a node, and the bill grows.

So the money is decided by requests. Usage only tells you how much of that money was wasted.

Requests usually get set once, early, by someone guessing. Nobody wants their service to be the one that ran out of memory, so the guess is generous, gets copied into the next Helm chart, and never gets revisited.

Step 1: Measure the gap

Start with a quick look from the command line. This needs metrics-server, which most managed clusters ship with:

kubectl top nodes kubectl describe node <node-name> | grep -A 8 "Allocated resources"

kubectl top nodes shows real usage. The "Allocated resources" section of describe node shows how much of the node is requested. A node at 90 percent requested and 25 percent used is the pattern you are looking for.

For the whole cluster, Prometheus with kube-state-metrics gives you the same comparison per namespace. Requested CPU:

sum by (namespace) ( kube_pod_container_resource_requests{resource="cpu"} )

CPU actually used:

sum by (namespace) ( rate(container_cpu_usage_seconds_total{container!=""}[5m]) )

And the same for memory, requested against the working set:

sum by (namespace) (kube_pod_container_resource_requests{resource="memory"}) sum by (namespace) (container_memory_working_set_bytes{container!=""})

Put requested and used side by side in one table and sort by the difference. In most clusters, a handful of namespaces account for most of the waste. Start there.

Step 2: Right size from real data, not from a guess

Right sizing means setting requests close to what the workload really needs, with sensible headroom. The key word is real: look at at least a week of history so you catch daily peaks and the weekly batch job.

For CPU, the 95th percentile over a week is a good starting point for the request:

quantile_over_time(0.95, rate(container_cpu_usage_seconds_total{namespace="shop", container="api"}[5m])[7d:5m] )

For memory, look at the peak instead, because running out of memory gets the container killed rather than slowed down:

max_over_time( container_memory_working_set_bytes{namespace="shop", container="api"}[7d] )

A few rules that keep this safe:

  • CPU request around the p95 of usage. CPU is compressible. If a pod needs more for a moment and the node has spare cycles, it gets them.
  • Memory request at the observed peak plus 20 to 30 percent headroom, and set the memory limit to the same value or a bit higher. Memory is not compressible. A container over its limit is OOM killed.
  • Change one service at a time and watch latency, restarts and OOM kills for a few days before moving on.
  • Watch out for startup spikes. JVM and some Node.js apps use far more CPU while starting. If startup gets slow after a cut, that's the reason.

Let the Vertical Pod Autoscaler do the math

If you don't want to write the queries yourself, the Vertical Pod Autoscaler can recommend requests from observed usage. It's an add on you install into the cluster. Run it in recommendation only mode so it never touches running pods:

apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: api-vpa namespace: shop spec: targetRef: apiVersion: apps/v1 kind: Deployment name: api updatePolicy: updateMode: "Off"

After it has watched the workload for a while:

kubectl describe vpa api-vpa -n shop

The Target value under the recommendation is what it would set. Treat it as a strong suggestion, compare it with your own percentiles, and apply it through your normal deployment process. One caution: don't let VPA change CPU or memory automatically on a workload where the Horizontal Pod Autoscaler scales on the same metric. The two will fight.

Step 3: Turn off what nobody is using

Oversized requests are the biggest leak. Idle resources are the second, and they are often easier to fix.

Dev and staging environments running all night. A staging namespace that nobody touches between 8 PM and 8 AM and all weekend is idle for roughly two thirds of the week. Scaling those deployments to zero outside working hours is one of the fastest savings you can get. A small CronJob running kubectl scale deployment --all --replicas=0 -n staging in the evening, and a matching one in the morning, is enough to start. Tools like kube-downscaler do the same with annotations.

Forgotten volumes. Persistent volumes outlive the apps that created them, and you keep paying for the disks:

kubectl get pv | grep -E "Released|Available"

A Released volume is no longer bound to any claim. If its reclaim policy is Retain, the disk still exists in your cloud account. Check what's on it, snapshot it if needed, and delete it.

Load balancers nobody uses. Every Service of type LoadBalancer usually creates a separate cloud load balancer with its own hourly price:

kubectl get svc -A | grep LoadBalancer

Old test services and services that should sit behind your shared ingress show up here more often than you'd expect.

Abandoned namespaces. Preview environments and experiments from months ago keep running quietly. List namespaces with their age and ask the owners of anything old whether it still needs to exist.

Step 4: Make the nodes follow the pods

Right sizing frees up space on nodes, but you only save money if the cluster actually removes nodes afterwards.

The Cluster Autoscaler removes a node when the pods on it can be moved elsewhere and the node has been underused for a while. A few things commonly block that:

  • Pods with the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
  • Pods using local storage, like emptyDir volumes, which the autoscaler won't evict by default in many setups
  • PodDisruptionBudgets that allow zero disruptions
  • Bare pods that aren't managed by a Deployment, StatefulSet or other controller

If your nodes never scale down, check those first.

On AWS, Karpenter goes a step further with consolidation. It removes empty nodes and also replaces several underused nodes with fewer or cheaper ones:

disruption: consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 1m

That snippet goes in the NodePool spec. Combined with right sized requests, consolidation is where the savings finally show up on the bill.

Use spot capacity for workloads that can handle interruption. Stateless services with several replicas, CI runners and batch jobs are good candidates. Spot capacity can be much cheaper than on demand, at the cost of being reclaimed at short notice, so keep databases and single replica services on regular nodes.

Step 5: Show each team what it costs

Waste comes back if nobody can see it. The most lasting fix is to make cost visible per team.

OpenCost, a CNCF project, runs in the cluster and breaks down cost by namespace, deployment, label and more, using your cloud provider's pricing. Install it, make sure every workload carries a team or app label, and share a simple weekly report: what each team requested, what it used, and what that cost.

You don't need chargeback or hard budgets to see results. In most teams, seeing "you requested 40 CPUs and used 6" next to a dollar figure is enough to start the right conversation.

A simple order to do this in

  1. Measure requested vs used per namespace.
  2. Pick the three biggest gaps and right size them from a week of data.
  3. Scale dev and staging to zero out of hours.
  4. Clean up released volumes, unused load balancers and abandoned namespaces.
  5. Make sure nodes actually scale down, with consolidation if you have it.
  6. Put cost per team in front of the teams.

Each step is small on its own. Together they often cut a large share of the compute bill without touching a single line of application code.

Takeaway

In Kubernetes, you pay for what you request, not for what you use. Measure the gap between the two, right size from real usage with headroom for memory, switch off what sits idle, and make sure your nodes shrink when your pods do. Then make the numbers visible, so the gap doesn't quietly grow back.

Keep reading

Let's stay in touch

I'm Amit Shukla, a software engineer working across DevOps, cloud and applied AI. Behind the Stack is where I write up the things I learn the hard way, with working code and honest tradeoffs.

If this saved you some time, a reaction or a comment helps more people find it.