gRPC Load Balancing in Kubernetes: Why One Pod Gets All the Traffic (and How to Fix It)

Your gRPC service gets busier, so you bump it from 2 replicas to 6. The Horizontal Pod Autoscaler says all 6 are running. But when you look at CPU, one pod is at 90% and the other five are almost idle. Latency doesn't improve at all.

Nothing is broken. This is how gRPC and Kubernetes Services behave together by default, and it catches a lot of teams the first time they move internal calls from REST to gRPC.

If you're still deciding between the two protocols, start with this gRPC vs REST comparison, which covers the trade-offs in streaming, payload size, browser support and debugging. This post assumes you've already picked gRPC and now need it to spread load properly.

Why one pod gets all the traffic

A normal Kubernetes ClusterIP Service balances at layer 4. kube-proxy (using iptables or IPVS) picks a backend pod when a TCP connection is opened, and every byte on that connection goes to the same pod.

With REST over HTTP/1.1, that's fine. Clients open many short connections, or a pool of them, so connections and therefore requests get spread across pods.

gRPC runs on HTTP/2, which works differently:

• A client opens one long-lived connection to the server.
• Every RPC is a separate HTTP/2 stream multiplexed over that one connection.
• The connection can stay open for hours or days.

So kube-proxy balances once, when the connection opens, and then all calls go down that pipe. If your client service has 3 pods, each opens one connection, and you'll have at most 3 busy server pods no matter how many you scale to. Pods added later get nothing until some connection drops.

client pod A ──── 1 HTTP/2 connection ────► server pod 1 (busy) client pod B ──── 1 HTTP/2 connection ────► server pod 1 (busy) client pod C ──── 1 HTTP/2 connection ────► server pod 2 server pod 3 (idle) server pod 4 (idle) 

The fix is to balance per request (per RPC), not per connection. There are three main ways to do it.

Fix 1: Client-side load balancing with a headless Service

gRPC clients have a built-in round_robin balancer. Give the client every pod IP and it opens a connection to each one and spreads RPCs across them.

To get every pod IP from DNS, create a headless Service (clusterIP: None). Instead of one virtual IP, DNS returns an A record for each ready pod.

apiVersion: v1 kind: Service metadata: name: user-service-headless spec: clusterIP: None selector: app: user-service ports: - name: grpc port: 50051 targetPort: 50051 

Then point the client at it with the dns:/// scheme and turn on round_robin.

Go:

conn, err := grpc.NewClient( "dns:///user-service-headless.default.svc.cluster.local:50051", grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithDefaultServiceConfig(`{"loadBalancingConfig": [{"round_robin":{}}]}`), ) 

Python:

import grpc channel = grpc.insecure_channel( "dns:///user-service-headless.default.svc.cluster.local:50051", options=[("grpc.lb_policy_name", "round_robin")], ) 

Without that setting, gRPC uses pick_first, which connects to the first address and sends everything there. You'd be back where you started.

The catch: the client resolves DNS when it starts and again when a connection fails. It doesn't keep polling. So when you scale from 3 to 6 pods, existing clients won't see the new pods for a while. Fix 2 handles that.

Good fit: a small number of services, all owned by your team, where you don't want to run a mesh.

Fix 2: Make connections expire with MaxConnectionAge

You can make the server close each connection after a set time. The client reconnects cleanly, re-resolves DNS, and picks up any new pods. With a plain ClusterIP Service, a reconnect also gives kube-proxy another chance to pick a different pod, so even without client-side balancing things even out over time.

The server sends an HTTP/2 GOAWAY frame, so in-flight RPCs are allowed to finish during the grace period. Clients don't see errors.

Go:

server := grpc.NewServer( grpc.KeepaliveParams(keepalive.ServerParameters{ MaxConnectionAge: 5 * time.Minute, MaxConnectionAgeGrace: 30 * time.Second, }), ) 

Python:

server = grpc.server( futures.ThreadPoolExecutor(max_workers=10), options=[ ("grpc.max_connection_age_ms", 5 * 60 * 1000), ("grpc.max_connection_age_grace_ms", 30 * 1000), ], ) 

A few minutes is a sensible starting point. Shorter values rebalance faster but cost more reconnects. If you have long-running streaming RPCs, set the grace period longer than your longest expected stream, or those streams will be cut off.

Fix 1 plus Fix 2 together is the simplest production setup that works without extra infrastructure.

Fix 3: A layer 7 proxy or service mesh

The other option is to put something in the middle that understands HTTP/2 and balances each RPC itself. Your application code doesn't change at all.

• Linkerd detects HTTP/2 and gRPC automatically and balances per request using latency-aware (EWMA) balancing. Inject the sidecar and the problem goes away. It's the lightest option if your only goal is fixing gRPC balancing.
• Istio uses Envoy sidecars that balance per request too, and adds retries, mTLS, traffic splitting and a lot more. It's more powerful and also more to operate.
• A standalone Envoy (or an ingress controller with gRPC support) in front of the service works when you don't want sidecars everywhere.

The trade-off is an extra network hop per call and one more system to run. If you already have a mesh for mTLS or observability, this is the obvious choice. If you don't, Fix 1 and Fix 2 are often enough.

There's also proxyless gRPC with xDS, where the gRPC library gets endpoint lists directly from a control plane such as Istio or Traffic Director. It's powerful but more advanced, and worth reading up on once you outgrow DNS.

Which fix should you use?

How to check it's working

Don't trust the replica count. Check the load on each pod:

kubectl top pods -l app=user-service 

The CPU numbers should be roughly even across pods. Better still, export a per-pod RPC counter (most gRPC Prometheus interceptors give you grpc_server_handled_total) and graph it by pod. If one line is far above the others, connections are still sticking.

A quick load test also helps. Run ghz against the service, scale it up halfway through, and watch whether the new pods start taking requests:

ghz --insecure \ --proto ./user.proto \ --call UserService.GetUser \ -d '{"id": 12345}' \ -c 50 -z 2m \ user-service.default.svc.cluster.local:50051 

Key takeaways

• Kubernetes ClusterIP Services balance connections. gRPC sends many RPCs over one connection, so load ends up uneven.
• Use a headless Service with the round_robin policy so each client talks to every pod.
• Set MaxConnectionAge on the server so clients reconnect regularly and pick up new pods.
• If you run a service mesh like Linkerd or Istio, it already balances per request, with no code changes.
• Check the load on each pod, not just how many replicas are running.

This is one of the costs that comes with gRPC's speed and streaming. For the full picture of what you gain and what you give up compared with REST, read gRPC vs REST: Which API Technology Should You Use? on OnlineToolkitHub.