NVIDIA GPU Operator on RKE2: Passthrough to Kubeflow (The Sports Car in the Garage)

Episode 7 of the homelab build series. The foundation is poured, the walls are up, the alarms are wired, the storage floor is down and the safe is locked. Now for the fun room: three NVIDIA GPUs, one in each machine, so GPU capacity stays highly available when any one of them is down, managed by the NVIDIA GPU Operator and wired into Kubeflow.

nvidia-smi side by side on the three GPU nodes, one per datacenter.

Introduction

Three GPUs. Three datacenters. Zero schedulable.

I had three NVIDIA cards in three machines, and Kubernetes swore I had none. I asked for one GPU in a test pod and got this:

0/21 nodes are available: 21 Insufficient nvidia.com/gpu. 

The Proxmox hosts could see them. lspci could see them. nvidia-smi on the host lit up. But to the cluster, three cards' worth of Ampere silicon were parked, idle and invisible. That's the most expensive kind of idle there is.

And the whole MLOps platform from the earlier episodes (Kubeflow, Katib, KServe, the training pipelines) was built to train and serve real models. You can't schedule what the scheduler doesn't know exists.

Fitting the cards took a Saturday. Getting the kernel module, the container runtime, the device plugin and the scheduler to agree took a lot longer, and taught me thirteen different ways a GPU can hide from Kubernetes.

Here's where it ended up. Three RTX A1000s, one in each "datacenter", which here means one in each of three mini PCs on a shelf. Lose a whole machine and two cards are still there. The NVIDIA GPU Operator builds the driver, wires up the runtime and advertises the cards on its own. And the part I'm proud of: when the whole lab loses power, all three cards come back and pass their CUDA self-test without me touching anything.

This is how I got there, every trap along the way, and the parts that still aren't done.

And at the end there's a real job on a real card: a 7-billion-parameter language model answering at about 30 tokens a second, against 4 on the CPU beside it.

Where this sits in the build

• Foundation: Terraform on Proxmox.
• Framing: Ansible building the five-master HA RKE2 cluster.
• Security: Kyverno, KubeArmor and Trivy, with Falco parked for now to save CPU.
• Workshop: Tekton, Harbor, SonarQube and Chains.
• Storage floor: Rook-Ceph and CloudNativePG.
• The safe: HA HashiCorp Vault.
• The sports car: this one.

This is where the cluster stops being general-purpose Kubernetes and gains an accelerator pool. Before the operator went in, the three GPU workers were scheduled like any other node, by CPU and memory, with no idea a GPU was attached.

The repo side is small: one namespace, one chart, one values file, and one ClusterPolicy the operator builds everything from.

kubernetes-addons/0-boostrap/gpu-operator/ namespace.yaml # gpu-operator namespace, privileged Pod Security gpu-operator-values.yaml # annotated Helm values; every trap fix lives here dcgm-custom-metrics.yaml # extra DCGM fields for the dashboard nvidia-dcgm-dashboard.yaml # Grafana dashboard examples/ # multi-GPU best-practice manifests 

The hardware in the garage

The cards: three NVIDIA RTX A1000s. Ampere GA107GL, 8 GB of GDDR6, 2,304 CUDA cores, about 50 W. It draws all its power from the PCIe slot, with no extra connector, which is exactly why it fits where a bigger card wouldn't. The node label says it plainly:

$ kubectl get node rke2-worker-dc1-gpu-a \ -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product} {.metadata.labels.nvidia\.com/gpu\.memory}MiB' NVIDIA-RTX-A1000 8188MiB 

The hosts: three Minisforum MS-01s. Intel i9-13900H, 96 GB of DDR5, and, rarely for a mini PC, one real half-height PCIe slot wired to the CPU: x16 in length, x8 electrically, which is plenty for this card. That slot is the whole reason this build exists.

Three cards, three machines: PCIe passthrough from Proxmox host to RKE2 worker.

That's 24 GB of GPU memory across the fleet. One 8 GB card runs a 4-bit 7B model with room to spare; I measure that further down. It's great for inference, fine for small fine-tunes, and marginal for training from scratch, which is a rented-A100 job anyway.

Why one card per datacenter instead of three in one box? The same answer as storage: failure domains. If a machine goes down, GPU work still has two healthy places to land. In a homelab, rack density is worth nothing and HA is worth a lot.

Why bother with Kubernetes when the GPU is right there on the host? Three reasons. Each project gets its CUDA version from an image tag instead of eight CUDA installs fighting each other. The runtime becomes a pinned image, so results are reproducible. And the scheduler is the only sane way to say "this job gets one card, that one gets another".

PCIe passthrough: from Proxmox to the VM

Each card is bound to vfio-pci on its Proxmox host and passed into exactly one VM. An Ansible playbook handles the nouveau blacklist, the vfio binding, IOMMU and the reboot. All three cards sit at the same address, 0000:01:00, which keeps the Terraform pleasingly uniform.

The passthrough itself is Terraform with the bpg provider, through a gpu_pci_hosts map per datacenter. One gotcha deserves its own sentence: raw hostpci needs the root@pam Proxmox user. A normal API token can't assign raw PCI devices, and the error doesn't tell you that.

cd terraform/environment/prod-rke2/bpg terraform plan -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu # 0 to add, 2 to change, 0 to destroy (in place: +machine=q35, +hostpci0) terraform apply -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu 

machine=q35 and hostpci0 only take effect after a full power cycle of the VM, not a soft reboot.

The best moment of the whole build came here. Once dc1 worked, I passed the cards through on dc2 and dc3 and rebooted them. The card appeared, the node got labelled, and the operator rolled the entire stack onto both nodes by itself. The first node took me twelve hours. The other two took about ninety seconds each, and I didn't touch them.

Three hardware notes that earned their place:

• Driver version is pinned to 570.133.20. The version stays put. What has to follow the kernel is the module, which the driver container rebuilds against whatever kernel the VM is running.
• No vGPU, no MIG. The A1000 supports neither.
• Secure Boot never gets a say. The GPU VMs boot with SeaBIOS, not UEFI, so the unsigned NVIDIA module loads without any key enrolment. The Proxmox hosts themselves do have Secure Boot on; it doesn't matter, because the module loads inside the VM. If your VMs use UEFI with Secure Boot, you'll have to sign the module or switch it off there.

What the GPU Operator actually does

What the GPU Operator runs: a tower where every floor waits for the one below.

One Helm release of nvidia/gpu-operator (chart v26.3.2) runs a set of DaemonSets from a single ClusterPolicy. Sticking with the garage:

The dependency chain is the thing to remember. NFD labels the nodes that have a GPU. Everything else only runs where that label is. So the driver, toolkit and device plugin land on the three GPU workers and nowhere else. No NFD label, no driver. No driver, no GPU.

And every other piece waits for the driver. When the bottom of that tower is broken, everything above it sits in Init. That gave me the rule that saved the most time: debug from the driver up. There's no point staring at the device plugin while the driver is still crash-looping.

What I deliberately don't run: the MIG manager (the A1000 has no MIG), the vGPU manager (that's for licensed shared vGPU), the sandbox and Kata pieces (the card is passed through once, at Proxmox), GPUDirect Storage (no hardware for it), and a standalone DCGM (the exporter brings its own, and two would fight over one socket).

The stack lives in the gpu-operator namespace with privileged Pod Security. The driver, toolkit and device plugin load kernel modules and touch host paths, and a restricted namespace blocks every one of them.

kubectl get pods -n gpu-operator mid-cascade: a wall of Init:0/1 and one very sad nvidia-driver-daemonset. The "before" shot.

From PCIe slot to scheduled pod

From PCIe slot to scheduled pod: once per node, then once per pod.

Once per node:

1. The worker boots with the GPU present as a PCI device.
2. NFD labels the node.
3. The driver pod compiles nvidia.ko for the running kernel and loads it.
4. The toolkit registers the nvidia runtime with containerd.
5. The device plugin tells the kubelet the node has nvidia.com/gpu: 1.
6. GFD adds nvidia.com/gpu.product=NVIDIA-RTX-A1000 and friends.
7. The DCGM exporter starts and metrics flow.

Then once per pod:

1. A workload asks for nvidia.com/gpu: 1.
2. Kyverno adds the taint toleration and the GPU node rule (more on that in the next section).
3. The scheduler finds a GPU node with a free card and binds the pod.
4. The NVIDIA runtime injects the device and driver libraries.
5. nvidia-smi inside the container sees its card.

All of it is automatic once the operator is in. That's the whole argument against a hand-rolled Ansible role that builds kernel modules per node and breaks on every kernel upgrade.

Keeping GPU pods on GPU nodes

Two bits of metadata mark these nodes, and you want both:

• A label so workloads can find them: nodepool=gpu, plus everything NFD and GFD add.
• A taint so ordinary workloads can't drift onto them and starve a GPU job: nvidia.com/gpu=true:NoSchedule.

$ kubectl get node rke2-worker-dc1-gpu-a -o jsonpath='{.spec.taints}' [{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"true"}] 

Normally that means every GPU pod needs a toleration, a node selector and a resource limit, and forgetting any one gives you a confusing failure. On this cluster a Kyverno policy, gpu-pods-to-gpu-nodes, does the first two for you. Any pod that asks for nvidia.com/gpu is mutated to tolerate the taint and require a nodepool=gpu node. So a GPU pod here needs exactly one line:

resources: limits: nvidia.com/gpu: 1 

The operator's own DaemonSets carry the toleration in the values file, and NFD's master also needs a control-plane toleration:

daemonsets: priorityClassName: system-node-critical tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule nfd: master: tolerations: - key: node-role.kubernetes.io/control-plane operator: Equal value: "" effect: NoSchedule 

Prove it works

The quickest proof is NVIDIA's own CUDA sample. It adds two vectors on the card and prints one line if the whole chain held: driver, runtime, device plugin, scheduler.

apiVersion: v1 kind: Pod metadata: name: cuda-vectoradd namespace: default spec: restartPolicy: OnFailure securityContext: runAsNonRoot: true runAsUser: 1000 seccompProfile: type: RuntimeDefault containers: - name: cuda-vectoradd image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL resources: limits: nvidia.com/gpu: 1 

Look at what isn't there: no toleration, no node selector, no runtimeClassName. The only GPU line is the limit. The securityContext blocks have nothing to do with the GPU; my default namespace enforces the restricted Pod Security Standard, so every pod there needs them.

$ kubectl apply -f cuda-vectoradd.yaml $ kubectl get pod cuda-vectoradd -o wide NAME READY STATUS RESTARTS AGE NODE cuda-vectoradd 0/1 Completed 0 16s rke2-worker-dc1-gpu-a $ kubectl logs cuda-vectoradd [Vector addition of 50000 elements] Copy input data from the host memory to the CUDA device CUDA kernel launch with 196 blocks of 256 threads Copy output data from the CUDA device to the host memory Test PASSED Done 

Sixteen seconds from apply to Completed, image pull included. The pod landed on a GPU worker without being told to, and you can see what Kyverno wrote into it on the way in:

$ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.tolerations}' [{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Equal","value":"true"}] $ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.affinity.nodeAffinity}' {"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"nodepool","operator":"In","values":["gpu"]}]}]}} 

Delete the pod when you're done. It has finished, so it isn't holding the card, but it's clutter.

kubectl logs cuda-vectoradd ending in Test PASSED. The hero shot.

The runtime: getting containerd to know about GPUs

containerd doesn't know what a GPU is. Something has to register an nvidia runtime that injects the device and driver libraries into containers. The classic way is to patch containerd's config.toml, and RKE2 regenerates that file from a template, throwing your patch away on every restart.

Two things make it work here. CDI is on, so the toolkit writes a device spec at /etc/cdi/nvidia.yaml that containerd reads at container start:

cdi: enabled: true nriPluginEnabled: false 

And the toolkit is pointed at RKE2's own containerd rather than the default one:

toolkit: env: - name: CONTAINERD_CONFIG value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml - name: CONTAINERD_SOCKET value: /run/k3s/containerd/containerd.sock - name: CONTAINERD_RUNTIME_CLASS value: nvidia - name: CONTAINERD_SET_AS_DEFAULT value: "true" 

The subtle bit is that CONTAINERD_CONFIG points at the generated config.toml, not the .tmpl. The toolkit re-applies its setup every time it starts, so when a reboot regenerates the config without nvidia, the toolkit simply puts it back. That's what makes it heal itself across reboots. Point it at the template and the live config sits there without a GPU runtime until someone restarts rke2-agent by hand.

Because nvidia is the default runtime, workloads here don't need runtimeClassName: nvidia. On a cluster without that setting they do, and that one line is what confuses people copying manifests between clusters.

Sharing the cards

Each A1000 advertises exactly one nvidia.com/gpu, so by default one pod owns the whole card.

The node labels confirm it: nvidia.com/gpu.replicas=1 and sharing-strategy=none on all three.

Time-slicing is a ConfigMap plus a patch to the ClusterPolicy:

apiVersion: v1 kind: ConfigMap metadata: name: time-slicing-config namespace: gpu-operator data: a1000: |- version: v1 sharing: timeSlicing: resources: - name: nvidia.com/gpu replicas: 4 # one A1000 looks like four schedulable GPUs 

kubectl patch clusterpolicies.nvidia.com cluster-policy --type merge \ -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"a1000"}}}}' 

Create the ConfigMap first, then patch. I ran the patch on its own once, and the device plugin's replacement pod sat in Init with configmap "time-slicing-config" not found, which took that node's card from one schedulable GPU to zero until I cleared the reference again. The DaemonSet replaces one pod at a time, so the rollout stalled there and the other two cards kept working.

Before you turn it on, remember it's oversubscription, not partitioning:

• No memory isolation. Four pods share 8 GB. Four 2 GB models fit; a fourth 3 GB model dies with CUDA out-of-memory.
• No fault isolation. One pod that wedges the card affects its neighbours.
• Sharing, not guarantees. Right for notebooks and bursty inference, wrong for anything with a latency target.

My plan is to time-slice dc1 and dc2 and leave dc3 whole for jobs that need all 8 GB.

How reliable is it?

Here's what the GPU tier survives, and the evidence for each:

And the honest limits. "Highly available" here means GPU capacity: a job running on a card dies with its machine and has to start again elsewhere. The operator itself is a single replica; if it's down, running GPU pods carry on and nothing new gets configured. A kernel upgrade on a GPU node still needs a driver rebuild (see the traps), there are no GPU alerts wired yet, and nothing caps how many cards one namespace can take. Those last three are on the list at the end.

kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator: three validators, one per datacenter, all Completed.

The traps, from the driver up

Thirteen ways a GPU hides from Kubernetes, by layer, from the driver up.

Thirteen of them, grouped by layer, in the order you should debug them. Most of them bit me. A few I only know from other people's write-ups, and those say so. Every piece waits on the driver, so a broken foundation makes everything above it look broken too.

The driver

The driver gets OOM-killed, exit 137. The log dies right after an nvidia-drm warning, then nvidia-smi: not found, and it crash-loops. It looks like a driver bug. It isn't: the driver container compiles the kernel module, running gcc against the kernel headers, and a 512Mi limit kills it mid-compile.

driver: resources: limits: cpu: "2" memory: 4Gi # 512Mi OOMs the kernel-module compile 

failed to install nvidia kernel module: distro 'ubuntu24.04' not supported. The driver image ships pre-built modules for a fixed list of distro and kernel pairs, and mine wasn't on it. Build from source instead:

driver: kernelModuleType: auto # open modules first, proprietary as fallback usePrecompiled: false # build from source, about 3 minutes on first boot 

nvidia: Unknown symbol __register_chrdev after a host kernel upgrade. The cached module was built for the old kernel. Force a rebuild:

kubectl -n gpu-operator rollout restart daemonset/nvidia-driver-daemonset 

The real lesson is a runbook entry, not a config change: a GPU node's kernel upgrade is a special operation, with a driver restart on the checklist. A plain power cut doesn't need it, because the kernel hasn't changed.

Missing kernel headers. The driver builds against the running kernel, so a kernel upgraded without linux-headers-$(uname -r) fails the build. The Ansible role for the GPU pool installs them, and the error is plain in the driver pod log.

The validator

nvidia-validator: command not found, exit 127. Version skew, and the error tells you nothing. Older charts shipped the validator as a separate image; the 26.x chart puts the binary inside the main operator image. Values written for one layout against a chart of the other produce a missing binary. Pin the chart (v26.3.2 here), deploy it from the NVIDIA repo rather than a stale vendored copy, and check what's actually running with helm list -n gpu-operator, not what your values file assumes.

The toolkit and runtime

no runtime for "nvidia" is configured. The RKE2 trap the generic docs miss. The toolkit looks for /run/containerd/containerd.sock; RKE2 runs its own containerd at /run/k3s/containerd/containerd.sock. The fix is the toolkit.env block above.

Failed to initialize NVML: Driver/library version mismatch inside a pod, while the host shows the same version. The toolkit had patched config.toml directly and RKE2 regenerated it. CDI or the generated-config setting fixes it.

KubeArmor's AppArmor profile blocks the toolkit. The same family of trap that once bit Cilium here: the toolkit can't write what it needs, containerd is silently never configured, and pods report the nvidia RuntimeClass as missing. If you hit it, run the toolkit container unconfined. Today it runs fine under its own KubeArmor-generated profile.

The toolkit writes the CDI spec and containerd never reads it. Pods schedule, but nvidia-smi hangs. CDI needs containerd 1.7 or newer, so check what your RKE2 release ships before you switch it on. I didn't hit this one: my containerd is 2.x.

Missing runtimeClassName: nvidia on a cluster where nvidia isn't the default runtime. The pod schedules, runs with no GPU injected, and looks exactly like broken hardware. Not an issue here because of CONTAINERD_SET_AS_DEFAULT, which is why it's worth knowing: the requirement depends on the cluster.

The device plugin and scheduling

Insufficient nvidia.com/gpu while three healthy GPUs sit idle. The driver and toolkit were running, but the device plugin was Pending:

0/21 nodes are available: 21 node(s) had untolerated taint {nvidia.com/gpu: true}, that the pod didn't tolerate. 

No device-plugin pod means nothing tells the kubelet about the card. The upstream chart ships the right toleration, but a trimmed values file copied from a blog about an untainted cluster quietly drops it. Same trap one layer down: NFD's master without its control-plane toleration never labels anything, so nothing else schedules either.

NFD takes a minute on first install. The operator comes up before NFD's first scan, so for about a minute it looks like the operator is running and nothing is happening. It isn't broken. Don't start changing things.

Telemetry

The DCGM exporter gets OOM-killed right after logging DCGM successfully initialized!. It loads its whole field set into memory and 256Mi isn't enough:

dcgmExporter: resources: limits: memory: 1Gi 

All of these fixes live in gpu-operator-values.yaml, .

Who holds a card, and how to get it back

This is the part I'd most have wanted to read, because it's invisible until a GPU job won't schedule.

nvidia.com/gpu can't be oversubscribed. Without time-slicing, a pod with nvidia.com/gpu: 1 owns that physical card for its whole life. Three cards means three GPU pods at once, full stop.

That makes long-lived notebooks the quiet problem. At one point two Kubeflow notebooks were each holding a whole A1000, one of them for three and a half months, while asking for 0.6 and 1.2 CPUs. A notebook that asks for less than one CPU and holds an entire GPU is the shape of the problem. It also quietly capped a Katib sweep at one GPU trial at a time. And they cost more than cards: together they used over half of their namespace's CPU quota, enough to make a daily training pipeline time out because its last job could never be scheduled.

The claim is on the pod, not the Notebook, the StatefulSet or the volume. So stopping a notebook frees the card. Kubeflow's "stop" sets an annotation, the controller scales to zero, and the pod goes away:

kubectl -n <profile> annotate notebook <name> \ kubeflow-resource-stopped="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite 

I sampled the free-GPU count every ten seconds while doing it:

[01] sts replicas=0 pod=Terminating FREE GPUs=1 [02] sts replicas=0 pod=gone FREE GPUs=2 

Look at the first line. The notebook was "stopped" by every API you'd think to check, and the card was still claimed, because the pod hadn't finished terminating. replicas=0 isn't proof a card came back. On a cluster where KubeArmor can wedge pods in Terminating, that matters. The reliable check counts from the pods and cross-checks the scheduler:

# what pods claim kubectl get pods -A -o json | jq '[.items[] | select(.status.phase=="Running" or .status.phase=="Pending") | .spec.containers[].resources.limits["nvidia.com/gpu"] // "0" | tonumber] | add' # what the scheduler has allocated on a node kubectl describe node <gpu-node> | awk '/Allocated resources/,/Events/' | grep nvidia.com/gpu 

As I write this, both GPU notebooks are stopped and all three cards are free.

Culling is on as a safety net. The notebook controller stops notebooks that have been idle for 24 hours. It stops, it never deletes: the volume and every file survive, and restarting is one click. What's lost is in-memory kernel state. A running cell counts as busy, so a long training run won't be culled. And the setting is global and not GPU-aware, so it catches abandoned notebooks; it doesn't manage contention. The better lever is not giving notebooks a GPU by default.

⚠️ One more surprise: nvidia.com/gpu isn't in the profile namespaces' ResourceQuota, and a quota only limits what it names. So GPU use per namespace is unbounded even though CPU and memory are capped.

Putting the GPUs to work

Making nvidia.com/gpu schedulable was the prerequisite, not the goal.

Who on the platform can ask for a card

Nothing on this platform needs a GPU to run. Every Kubeflow component is a CPU service; what changes is the work users launch through them. Here's each door to a card, and whether anything has walked through it yet:

Everything else in the data platform (Kafka, Airflow, MLflow, the databases) never touches a card and shouldn't.

So the honest summary is two real users so far: notebooks, which took cards and did nothing with them, and one benchmark pod, which took a card and drove it at 98%.

Hyperparameter search with Katib

Katib runs an Experiment that launches trials as pods, each reporting a metric back through an injected sidecar. Getting it to run at all taught me more than the GPUs did.

It couldn't run in any Kubeflow profile namespace. An Experiment sat at Created with zero trials, and the UI only said "Couldn't find any successful Trial." The real error was two levels down, on a ReplicaSet:

pods "<exp>-tpe-..." is forbidden: minimum memory usage per Container is 16Mi, but request is 10Mi 

Katib asks for 10Mi of memory for its suggestion pods, and the Kyverno-generated LimitRange in every profile namespace has a 16Mi floor. Eleven of twelve algorithms had no explicit resources, so eleven of twelve couldn't schedule. Fixing those exposed the same problem on the trial pods, because the metrics sidecar is injected by a webhook and its default never shows up in the template. The fix is to raise Katib to meet the floor, not lower the floor:

runtime: suggestions: - algorithmName: tpe image: ghcr.io/kubeflow/katib/suggestion-hyperopt:v0.19.0 r...