GPU clusters treat a GPU as healthy or broken. Real GPUs also end up slow: a lowered
power limit, a clock lock, heat. That middle state is expensive in a specific way: in
synchronous distributed training every step waits for the slowest GPU, so one slow GPU
slows every GPU in the job, and you pay for all of them.

The usual answer is to remove the slow GPU. But "slow" depends on the work. So I built
GPUFit, an open-source tool that grades every GPU per workload (training, fine-tuning,
inference) and uses that grade for scheduling in Kubernetes: a degraded GPU is kept out of
synchronous training and keeps serving inference.

Everything below was measured on real NVIDIA GPUs on AWS. Total spend: about $8.50.

The same fault hurts workloads very differently

One A10G, power limit lowered from its default 300 W, three runs each:

Power limit Heavy matrix multiply (training-like) Memory copy (inference-like)
250 W 0.94 1.00
150 W 0.68 1.00
100 W 0.35 1.00

At 150 W the GPU lost a third of its speed for compute-heavy work and nothing for
memory-bound work. A "healthy/unhealthy" flag can't express that. A per-workload grade can.

How GPUFit works

  • An agent on each GPU node turns GPU health (from nvidia-smi) into a score per workload. Training uses the weakest component, because one straggler sets the pace.
  • A controller moves a node with a GPU that's unfit for training into an inference pool, using Kubernetes DRA device taints (Dynamic Resource Allocation, stable in Kubernetes 1.37). Training claims don't tolerate the taint; inference claims do. No custom scheduler, and it works with any DRA driver, including NVIDIA's.
  • Hysteresis: a GPU must stay degraded for a while before it moves, and stay healthy for longer before it comes back, so noisy GPUs don't flap.
  • Observe mode (the default) changes nothing and reports what it would do, plus kubectl gpufit report: training time on degraded GPUs and the estimated waste. Enforce mode applies it. Optional rescue moves running training jobs that opted in off a degraded GPU; they resume from their checkpoint.
  • kubectl gpufit why <pod> explains a Pending GPU pod. The scheduler says "cannot allocate all claims" whether the cluster is busy or the GPUs are unfit; this tells them apart.

Install: helm install gpufit oci://ghcr.io/praveen9161/charts/gpufit --version 0.1.0 -n gpufit --create-namespace

Result 1: -25% cost per useful GPU-hour

Two A10Gs, k3s, NVIDIA's DRA driver. GPU B's power limit cut to 150 W.

  • Without GPUFit, Kubernetes put a 2-GPU training job on A+B. It ran at 0.84 of its healthy speed: one slow GPU dragged the healthy one down.
  • With GPUFit, B moved to the inference pool. Training got the healthy GPU (1.00) and B served inference at 0.98 of a healthy GPU.
Useful work (healthy-GPU equivalents) Efficiency $ per useful GPU-hour
Without GPUFit 1.49 74% $1.35
With GPUFit 1.98 99% $1.02

The honest caveat: with only two GPUs and no spare, the training job itself got slower
(one GPU instead of two). The gain is that almost no paid GPU time was wasted. In a bigger
cluster, training would get a healthy spare.

Result 2: rescuing a job that's already running

Four A10Gs, GPUFit installed with its Helm chart. A 2-GPU training job (400 steps,
checkpoint every 50); at step 100 one of its GPUs drops to 150 W.

Run Job time
Healthy 901 s
Degraded, GPUFit in observe mode (job stays as a straggler) 1001 s
Degraded, GPUFit in enforce + rescue mode 987 s

What the rescue did: 32 s after the power cut the node left the training pool; 36 s later
(a 30 s grace period) GPUFit evicted the job; 22 s later both pods ran again on healthy GPUs
and resumed from step 100.

A 14-second win is small, and that's the interesting part. The straggler lost 13% of every
remaining second; the rescue cost about 86 s, fixed. So rescue pays off once more than
about 10 minutes of work remain after the GPU degrades (an estimate from these
measurements; about 16 minutes with production timers). Real training jobs run for hours.

What didn't work (yet)

  • The waste estimate is too pessimistic. GPUFit's report estimates speed from the score (score 50 = half speed). The real job ran at 0.87, so the estimate was 3.8x the measured waste. Scores from hardware settings are a trigger, not a speedometer.
  • Small scale. 2-4 GPUs, synthetic workloads, one kind of induced fault. A straggler slows every GPU in its job, so the loss grows with job size; I haven't measured beyond 2-GPU jobs.
  • Network slowdowns aren't covered. In the FALCON study (Alibaba, HKUST), network slowdowns were 7.2x more common than GPU compute slowdowns.

Things that bit me

  • k3s' default pod network (10.42.0.0/16) overlapped my VPC (also 10.42.0.0/16). Nodes silently routed each other's real addresses into the pod network.
  • NVIDIA's DRA kubelet plugin only runs on nodes labelled nvidia.com/gpu.present=true, normally set by GPU feature discovery, which I hadn't installed: zero GPUs published.
  • helm uninstall left my taints behind, so training would have avoided those GPUs forever. The fix (a pre-delete hook) first failed too: the controller was still running and re-applied the taint one second after cleanup. Now the hook stops the controller first.
  • helm upgrade --reuse-values turns a null value override into an empty string. My agent quietly stopped scheduling on the test cluster.
  • An evicted pod still reports Running while it shuts down. My first rescue timeline said "running again after 0 s".

How this compares

Others detect slow GPUs well: NVIDIA's nvidia-resiliency-ext scores each GPU inside
training, ByteDance's MegaScale and FALCON (Alibaba, HKUST) find stragglers across thousands of GPUs,
NVIDIA's DRA driver taints GPUs with hard faults, and HeaRank (KDD 2026) steers GPUs likely
to fail toward interruption-tolerant work. What I couldn't find in the open is the next step:
grading by measured performance per workload and keeping the slow GPUs earning on
inference through standard Kubernetes DRA, with the cost effect measured.

What's next

Use the real per-GPU speed from the training jobs (via NVIDIA's own tooling) instead of
estimating it, grade GPUs separately for LLM prefill and decode so degraded GPUs can feed
decode pools, catch network slowdowns, and measure at 8-32 GPUs.

Code, every raw result, and a cost calculator for your own fleet:
https://github.com/praveen9161/gpufit (Apache-2.0).
Calculator: https://praveen9161.github.io/gpufit/calculator/

Feedback welcome, especially if you run GPU fleets: how often do your GPUs end up slow but not broken?