Brownfield Kubeflow on RKE2: Swapping Dex & SeaweedFS for Keycloak & Ceph RGW

Upstream Kubeflow assumes a greenfield cluster and ships its own Dex, SeaweedFS and MySQL, with its own oauth2-proxy in front. My homelab already had better versions of most of those. This is the **brownfield* deploy on an HA RKE2 cluster: every reference component I ripped out (and the one I haven't yet), what I wired in its place, and the auth chain that ate most of the debugging time.*

The Kubeflow Central Dashboard, logged in through Keycloak SSO (not Dex), with the whole platform iframed into one tab.

Introduction

When the platform assumes it's the only thing in the cluster

Kubeflow was designed as if it would have the cluster to itself, and it still shows. Point the reference installer at an empty cluster and it works well: nine controllers, four web UIs, two workflow engines, an inference layer and a per-user namespace system, all iframed into one dashboard and running in twenty minutes.

Point it at a cluster that already has an identity provider, an object store, a database fleet, a service mesh and an SSO layer guarding ten other apps, and you're in for a fight. Kubeflow never asks what you already run. The v1.11 multi-user install brings Dex for identity (behind its own oauth2-proxy, wired into Istio as an external authorizer), SeaweedFS for object storage, MySQL for state, and a mesh-wide trust model that doesn't match any real cluster I know of.

That's the gap between a greenfield deploy and a brownfield one. Greenfield, you accept whatever the platform brings. Brownfield, the platform moves into a neighbourhood that was there first. Mine already had Keycloak issuing every realm, Rook-Ceph serving S3 to half the stack, CloudNativePG holding everyone's relational state, and oauth2-proxy in front of ten apps.

I could run two of every backend, or rewrite Kubeflow's integration layer to talk to what was already in the building. I picked the rewrite. It cost roughly six weekends and a kustomize overlay per component, and the platform now fits the cluster it runs on.

Here's the full list: every reference component upstream Kubeflow ships, what I put in its place, and why the stock one didn't fit.

The rest of this piece walks through those swaps, the "huge CRD" trap that breaks client-side apply, the per-Profile governance Kyverno generates, and the Auth Chain Saga, where RBAC: access denied turned up at five different layers over several days before login finally worked.

Why Kubeflow, and which release

I pinned to v1.11.0 of the kubeflow/manifests repo, the most recent immutable release tag when I started. Don't be fooled by releases/release-1.11/ on master: it's a release-process tracker (a README about dates and the team). The real installer entry point is example/kustomization.yaml, and that differs between master HEAD (a moving target) and v1.11.0 (immutable). I wanted a build I could reproduce, so I took the tag.

Most of that time went on the auth chain (the Auth Chain Saga and the notebook routing pivot below). The heavy KFP overlay and the Model Registry integration were the runners-up.

The honest head-to-head: Kubeflow vs SageMaker/Vertex, Flyte, Metaflow, Ray, and MLflow

People ask me "should I use Kubeflow or MLflow?" a lot, and the question doesn't quite work. It's like asking whether to buy a kitchen or a knife. Kubeflow is a nine-component platform, and almost everything people compare it to replaces exactly one of those components. To compare fairly you have to sort the field by the layer each tool occupies:

1. Whole self-hosted platform (notebooks + pipelines + training + HPO + serving + registry + multi-tenant portal): Kubeflow, plus the open all-in-ones MLRun and Determined AI.
2. Managed cloud platform (all of the above, on someone else's hardware): AWS SageMaker, GCP Vertex AI, Azure ML.
3. Pipeline / orchestration only: Flyte, Metaflow, Airflow, Prefect, Dagster. These compete with KFP, which is one of the nine boxes.
4. Experiment tracking + registry only: MLflow, Weights & Biases, ClearML. A component, not a platform (I run MLflow inside this Kubeflow).
5. Serving only: KServe (which is part of Kubeflow), Seldon, BentoML, Triton.
6. Distributed compute only: Ray (KubeRay), which runs alongside Kubeflow rather than replacing it.

In other words, MLflow, Seldon and Ray are things you run inside Kubeflow, not alternatives to it. The only real peers to the whole platform are the managed clouds and a couple of open all-in-ones:

Where each one wins:

• SageMaker / Vertex / Azure ML: the best managed DX, zero platform ops, elastic GPU infra, wired into cloud data and IAM. If you already live in a cloud and care more about time-to-value than control, cost and data sovereignty, managed wins clearly. Kubeflow's answer is the flip side: no cloud lock-in, no per-GPU-hour rental markup, data never leaves my rack, and it runs on hardware I own. Think of it as a self-hosted SageMaker for people who want to own the platform.
• Flyte: honestly, a better pipeline engine than KFP. Strongly-typed Python, first-class versioning and caching, cleaner local dev. If pipelines are the whole job and you don't need notebooks, serving, HPO or a multi-tenant portal, Flyte is the nicer tool. Kubeflow wins on breadth: the nine boxes plus the portal that puts them in one tab.
• Metaflow: a pleasant Python DX from Netflix, aimed at data scientists who don't want to think about infrastructure. It covers orchestration and tracking, not serving, HPO or multi-tenancy, and its Kubernetes support is thinner than its AWS support.
• Ray (KubeRay): the best tool for distributed compute: training big models, hyperparameter tuning, RL, distributed data. It doesn't replace Kubeflow, it sits under it (Katib overlaps Ray Tune; KServe can serve Ray). If your bottleneck is Python-native distributed compute, run Ray alongside Kubeflow.
• MLflow / W&B: the best experiment tracking and registry around. That's why MLflow is a tile in my Central Dashboard rather than something I picked instead of Kubeflow.
• Seldon / BentoML / Triton: serving specialists. KServe covers most cases, but for advanced inference graphs or GPU-optimized Triton runtimes they can beat it on serving alone.

Why Kubeflow here, then? It's the one open-source platform that gives me all nine boxes (Jupyter notebooks, KFP pipelines, Trainer V2, Katib HPO, KServe serving, Model Registry and per-user Profiles) on my own RKE2 + GPU hardware, integrated with the stacks I already run (Keycloak, Ceph, CNPG, Istio), with no cloud lock-in. I couldn't find anything else in open source that complete and also self-hosted.

The catch is complexity. Kubeflow has a lot of moving parts, an auth chain that took several days to untangle, and an installer that assumes a greenfield cluster. If all you need is pipelines, Flyte has a fraction of the operational surface. If you'll pay for managed, SageMaker takes the ops away entirely.

Kubeflow only makes sense when you want the full platform, self-hosted, on your own hardware, and you're prepared to pay the complexity cost once. If you don't need all that, a specialist tool or a managed service on a credit card is the better call.

The cluster Kubeflow is landing on

The facts that matter:

• RKE2 with five masters and a mix of worker pools (app, data, GPU). HA control plane via kube-vip. The whole lab powers off every night and cold-boots in the morning, which matters more than I expected (see the cold-boot race below).
• Cilium in eBPF mode, with Hubble enabled.
• Istio 1.30 as the service mesh. The default mode is PERMISSIVE, and per-namespace PeerAuthentication CRs flip to STRICT where it matters. istio-system's default stays PERMISSIVE because the API server has to reach admission webhooks in injected namespaces without holding a client cert.
• Rook-Ceph for storage, with three SCs: ceph-block (RBD, RWO), ceph-filesystem (CephFS, RWX), ceph-bucket (RGW, S3 via OBC).
• CloudNativePG for relational state. The MLOps cluster postgres-mlops-cnpg-cluster hosts kubeflow_metadata_db, ready for Kubeflow Pipelines but not in use yet.
• Vault + Vault Secrets Operator, auto-unsealed by a separate transit Vault in its own vault-transit namespace (Shamir, 1 share).
• Keycloak as the OIDC provider for every app. Realms and clients are managed by Terraform; there are no Keycloak operator CRDs on this cluster.
• ArgoCD, driven by AppSets. Each major area (mlops-stack, kubeflow-stack, cicd-stack, security-stack, monitoring-stack) has its own AppSet generating a child App per component.
• Kyverno for admission policy and generate rules. At the time, the cluster-wide make-configmaps-immutable policy was the one that bit hardest on this build.

The kubeflow-stack.yaml AppSet in kubernetes-addons/0-georgehomelab-argocd-projects/2-argocd-applicationsets/ phases the rollout. Sync waves run from 1 (foundation) to 8 (the SSO front door), across 24 child Applications. Each component is its own Application, so I can debug it in isolation.

Kubeflow on RKE2, what I replaced from upstream: the cluster's Istio gateway, oauth2-proxy as a reverse proxy and Keycloak in front of the Central Dashboard, its six apps, and a foundation of GPUs, bundled MySQL, Ceph object storage and CloudNativePG, each marked replaced, upstream or added

Kubeflow as deployed: oauth2-proxy → Central dashboard → apps, MLflow beside it, and every swap from upstream tagged.

The ArgoCD UI filtered to kubeflow-*: the 24 child Applications from the kubeflow-stack AppSet, all Synced/Healthy, in sync-wave order.

Prerequisites first

Phase 0 is everything that has to exist before Phase 1's AppSet even sees the cluster. Five things:

1. The CNPG kubeflow_metadata_db Database CR. It lives in kubernetes-addons/Data-platform-stacks/mlops-cnpg/postgres-mlops-cnpg-cluster.yaml next to the other MLOps databases. The owner is the kubeflow Postgres role, with its password in Vault. Applying the Database CR creates the database on the already-running CNPG cluster. It's there for the MySQL → CNPG move, which hasn't happened yet (see the KFP overlay section).

2. Vault paths, populated by Terraform. Three matter:

   • secret/homelab/kubeflow/kubeflow-metadata-db: {username: kubeflow, password: ...}
   • secret/homelab/kubeflow/oauth2-proxy: {client-id, client-secret, cookie-secret} for the Keycloak kubeflow realm client.
   • secret/homelab/kubeflow/centraldashboard-session (optional): the Central Dashboard session secret. Terraform writes these from .secrets.auto.tfvars. No human writes Vault paths by hand.

3. The Keycloak kubeflow realm. Terraform creates the realm, its Google and GitHub identity providers and the oauth2-proxy client, and it reads that client's secret from the same Vault path oauth2-proxy uses, so the two can't drift. Before the SSO front door goes in, I check two things directly: the realm's .well-known/openid-configuration returns 200 with a valid issuer, and the Vault path holds all three keys.

4. Google OAuth client. Add kubeflow.georgehomelab.com/oauth2/callback to the authorized redirect URIs on the existing OAuth client.

5. DNS and a dedicated certificate. kubeflow.georgehomelab.com → 10.0.1.21 (the kube-vip ingress VIP), managed in Cloudflare. Kubeflow gets its own cert-manager Certificate (kubeflow-georgehomelab, issued via Let's Encrypt DNS-01), which writes the kubeflow-georgehomelab-tls-cert Secret the Kubeflow Gateway serves.

With those five in place, Phase 1 can fire.

Laying the foundation

Phase 1 is five Apps at sync waves 1 and 2.

kubeflow-namespace creates the kubeflow and kubeflow-system namespaces with pod-security.kubernetes.io/enforce: privileged. Upstream uses restricted, and my cluster default is also restricted. But every istio-injected namespace needs privileged, because istio-init runs with CAP_NET_ADMIN + CAP_NET_RAW to program iptables. Without privileged, istio-init fails admission and every pod in the namespace hits FailedCreate with no obvious cause. CNPG's initdb Job hits the same trap.

kubeflow-cert-issuer installs a selfSigned ClusterIssuer for Kubeflow's internal admission-webhook certs. Public TLS doesn't use it; that comes from the dedicated Let's Encrypt certificate.

kubeflow-roles holds the kubeflow-admin / kubeflow-edit / kubeflow-view ClusterRole aggregates. The Profiles controller binds them in per-user namespaces when it reconciles a Profile CR.

kubeflow-istio-resources is the dedicated kubeflow-gateway Gateway, which I put in istio-system rather than in the kubeflow namespace where upstream puts it. It binds kubeflow.georgehomelab.com specifically and serves the kubeflow-georgehomelab-tls-cert Secret, so I can rotate the Kubeflow cert without touching anything else.

The plan was failure-domain isolation from the shared homelab-gateway, and that only half worked. homelab-gateway's *.georgehomelab.com server overlaps this host on the same ingress gateway port 443, and Istio picks one per ingress replica. It showed up as intermittent 404s: curl worked, the browser didn't. The front-door VirtualService now binds both gateways, so the route exists whichever one Istio picks.

The same App later picked up the notebook ext_authz policy and its redirect EnvoyFilter (see the notebook routing section), but in Phase 1 it's just the Gateway and the RBAC aggregates.

kubeflow-kfp-infrastructure holds KFP's prerequisites. It creates the kfp-artifacts ObjectBucketClaim (Rook-Ceph provisions a bucket on RGW with its own access keys), plus the VaultAuth + VaultStaticSecret that sync the CNPG kubeflow user's credentials into a kubeflow-metadata-db Secret in the kubeflow namespace.

Those credentials are ready and unused. The ml-pipeline Deployment that lands in Phase 3 pulls the OBC-produced kfp-artifacts-bucket Secret via envFrom, but still reads its database credentials from the bundled MySQL's mysql-secret (I checked live). The same App also carries a daily backup CronJob that dumps both bundled MySQL servers (KFP and Katib) and ships them to a kubeflow-db-backups bucket on RGW. I only added it after noticing that neither database had a backup at all.

After Phase 1 there are no application pods yet, just CRDs, OBCs, Secrets and the gateway. That's on purpose. Phase 1 only lays the substrate.

Knative Serving and Eventing

Phase 2 is Knative. Serving is a KServe dependency (KServe's InferenceService Pod is built on Knative's Service CR). Eventing is in for completeness, in case a later workload wants Trigger / Broker to fan events out.

The deploy itself is uneventful; upstream's overlays land cleanly. The trouble is the CRD spec.conversion blocks. Some Knative CRDs (pingsources.sources.knative.dev, eventtypes.eventing.knative.dev) ship with webhookClientConfig set but spec.conversion.strategy missing. ArgoCD's ServerSideDiff dry-run rejects that with "Required value / Forbidden", and the App sits in ComparisonError, stuck on Unknown.

The fix goes in the AppSet template: argocd.argoproj.io/compare-options: ServerSideDiff=false. Falling back to the legacy 3-way merge skips the dry-run. You lose mutation-webhook injection in the diff, which I can live with because those webhooks are already in the broader ignoreDifferences block.

The second Knative trap: the controller rewrites the entire .webhooks[].rules[] array at runtime. It consolidates per-resource entries into wildcards, adds status subresources and uses apiVersion *, while the source manifest's rules block is a stub. Result: permanent OutOfSync. The fix is ignoreDifferences on the whole .webhooks array, scoped to the named Knative webhooks (webhook.serving.knative.dev, validation.webhook.serving.knative.dev).

A third one was on the risk register but never bit: the cluster-local-gateway. Knative needs an internal Istio gateway for in-cluster routing, and the homelab already had one in istio-system from an earlier deploy (30+ day uptime, helm-managed). Installing it again from the Knative manifests would fight the existing helm field-manager, so the Knative phase leaves the cluster-local-gateway out and relies on the one that's there.

The operators

Phase 3 is seven Apps at sync waves 4 and 5: admission-webhook, trainer, katib, pipelines (KFP), kserve, kserve-models-web-app and model-registry.

admission-webhook is the PodDefaults admission controller. It mutates pods at admission to inject volumes, env vars and secrets. Tiny footprint, no surprises, and it turns out to be the piece users notice most (more on that in the governance section).

trainer is Kubeflow Trainer V2 (formerly Training Operator V2). In 1.11 it's pure backend with no UI: the TrainJob / TrainingRuntime / ClusterTrainingRuntime CRDs and a controller manager. Users create TrainJob CRs from notebooks with the Python SDK (kubeflow.trainer).

Trainer has a kustomize trap. The upstream overlay sets app.kubernetes.io/part-of: kubeflow-training on both the Deployment's selector and the pod template. My overlay's labels: block (which stamps app.kubernetes.io/part-of: kubeflow-stack) with includeSelectors:false includeTemplates:true only touches the template, so selector and template disagree and the apiserver rejects the Deployment. The fix was to drop the labels: block from the trainer overlay.

katib is the hyperparameter optimization service: Experiment / Suggestion / Trial CRDs, a controller, a UI and a bundled MySQL backend. I left that MySQL alone. Katib only stores short-lived experiment metadata, and I saw no reason to put it on CNPG.

kserve and kserve-models-web-app are the inference layer. KServe's controller ships ~30 CRDs (InferenceService, InferenceGraph, ClusterServingRuntime and so on), and the Models Web App is a UI for listing and configuring InferenceService CRs. model-registry gets its own section further down.

The KFP heavy overlay

This is the most invasive overlay in the whole deploy. Upstream's applications/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user overlay assumes:

1. SeaweedFS for KFP artifacts: a bundled S3-compatible store that upstream exposes under the old minio-service Service name, so KFP's MinIO-era config keeps working. (Older releases shipped MinIO itself.)
2. MySQL for the ml-pipeline metadata database: bundled Deployment, Service and PVC.
3. A mlpipeline-minio-artifact Secret with hardcoded minio / minio123 creds.

I deleted the SeaweedFS workloads and replaced the Secret. MySQL is still running (more on that below).

SeaweedFS → RGW. Every KFP component that talks S3 now goes straight to RGW's public TLS endpoint, s3.georgehomelab.com:443. There are five of them:

• ml-pipeline (the API server) and cache-server, through env vars;
• pipeline-install-config (minioServiceHost);
• the Argo artifactRepository;
• the pipelines profile-controller's sync.py, through AWS_ENDPOINT_URL.

The first version of this overlay worked differently. It turned upstream's minio-service into an ExternalName pointing at the in-cluster RGW Service on plain port 80. That broke the day RGW went TLS-only and port 80 disappeared.

The ExternalName is still there, now pointing at s3.georgehomelab.com:443, but it's vestigial. Its only job is to stop upstream's minio-service, whose selector targets the SeaweedFS pod I deleted, from dangling. Nothing should depend on it. The minio-service.kubeflow* names were dropped from RGW's hosting.dnsNames, so a path-style request through that alias gets a 403.

Host and port alone aren't enough. ml-pipeline and cache-server read OBJECTSTORECONFIG_* straight from their env and ignore the ConfigMap's minioServicePort. Point them at port 443 without turning TLS on and they speak plain HTTP to a TLS port, and ensureBucketExists dies with upstream connect error or disconnect/reset before headers. The patch sets all three:

- name: OBJECTSTORECONFIG_HOST value: "s3.georgehomelab.com" - name: OBJECTSTORECONFIG_PORT value: "443" - name: OBJECTSTORECONFIG_SECURE value: "true" 

The artifact Secret. Upstream ships mlpipeline-minio-artifact with minio/minio123 in it. If you leave that Secret in the kustomize render, VSO's overwrite: true only merges. The placeholder keys survive, KFP reads accesskey=minio, and RGW answers "access key ID does not exist".

So the overlay deletes the upstream Secret with $patch: delete, and VSO creates it fresh from Vault with only the two keys KFP reads, accesskey and secretkey. The KFP Deployments reference the Secret by name, so they don't care who created it.

MySQL → CNPG (planned, not live). The cluster still runs the bundled mysql Deployment, and pipeline-install-config still says dbType=mysql. Upstream ships no multi-user Postgres overlay and the data is small, so the daily dump to RGW closed the real gap (no backups at all) first.

The CNPG database and its credentials are provisioned, and the patch below is the plan. The ml-pipeline, cache-server, metadata-grpc-deployment, ml-pipeline-persistenceagent, ml-pipeline-scheduledworkflow, ml-pipeline-ui and ml-pipeline-viewer-crd Deployments all take the database from env vars, and each gets patched to use the CNPG-provisioned creds:

env: - name: DBCONFIG_USER value: kubeflow - name: DBCONFIG_PASSWORD valueFrom: secretKeyRef: name: kubeflow-metadata-db key: password - name: DBCONFIG_HOST value: postgres-mlops-cnpg-cluster-rw.mlops.svc.cluster.local - name: DBCONFIG_PORT value: "5432" - name: DBCONFIG_NAME value: kubeflow_metadata_db 

The kubeflow-metadata-db Secret is VSO-synced from secret/homelab/kubeflow/kubeflow-metadata-db. Once this lands, the bundled MySQL Deployment goes via $patch: delete.

The istio init race. ml-pipeline-api-server calls glog.Fatalf on its first S3 connection failure, with no retry. Without proxy.istio.io/config: holdApplicationUntilProxyStarts: true, the apiserver gets "connection refused" because istio-proxy hasn't opened its outbound listeners yet, the app exits, and you have a CrashLoop. The annotation makes the main container wait for the sidecar's readiness probe. The same trap comes back with Model Registry's golang-migrate flow.

The cold-boot race. The lab powers off every night, so every morning Kubeflow starts from nothing. KFP's scheduler came up before ML Metadata, saw that the daily run's slot had been missed, and fired it straight away. The run's first step needs ML Metadata, so it failed before doing any work. I added an init container to the scheduler that waits for ML Metadata, MySQL, the cache server, Kyverno and a healthy API server, then gives everything two more minutes.

The first version of that wait didn't wait at all. It opened a TCP connection to each Service, and inside the mesh that always succeeds: the pod's own Envoy sidecar accepts the connection, whether or not anything is behind the Service. I only caught it by pointing the check at a Service with no endpoints and watching it pass. The fix is traffic.sidecar.istio.io/excludeOutboundPorts on the scheduler pod, so those checks go straight to the real Service. On the next cold boot it held the scheduler for 220 seconds while ML Metadata came up, and the run that followed succeeded.

template: metadata: annotations: traffic.sidecar.istio.io/excludeOutboundPorts: "8080,3306,443" 

A KFP run's artifact panel showing an s3://kfp-artifacts/... URI (served by Ceph RGW, not SeaweedFS), or the Ceph dashboard → Object Gateway → Buckets view with kfp-artifacts listed.

The "huge CRD" trap

Kubeflow CRDs are big. clustertrainingruntimes.trainer.kubeflow.org weighs ~280KB, trainjobs.trainer.kubeflow.org ~220KB, and the InferenceService and Knative ones aren't far behind.

The Kubernetes apiserver caps metadata.annotations at 262144 bytes (256KB). ArgoCD's default CSA-to-SSA migration writes a kubectl.kubernetes.io/last-applied-configuration annotation during the handover, and on these CRDs that annotation alone is >256KB. You get metadata.annotations: Too long: may not be more than 262144 bytes.

I stacked two fixes:

1. Per-CRD SSA via a kustomize patch. Put the argocd.argoproj.io/sync-options: ServerSideApply=true annotation on each problem CRD. ArgoCD then skips CSA and goes straight to SSA, which doesn't write the annotation.
2. App-level ClientSideApplyMigration=false. This turns the migration off entirely; it's set in the AppSet template's syncOptions. The CRD becomes argocd-controller-owned on first sync and stays SSA-managed.

Workspace UIs

Phase 4 is six Apps at sync wave 6: notebook-controller, jupyter-web-app, volumes-web-app, tensorboard-controller, tensorboards-web-app and pvcviewer-controller.

The web apps are SPAs and the controllers are headless. Between them, users can:

• spawn JupyterLab notebooks from the Jupyter UI;
• browse and create PVCs in the Volumes UI;
• launch TensorBoard instances against TensorBoard log dirs;
• view files in a PVC through the PVCViewer's per-PVC file browser.

Two integration points matter:

• Every UI reads a USERID_HEADER env to find out who's making the request. Upstream's default is kubeflow-userid; I override it to X-Forwarded-Email (the Auth Chain Saga explains why).
• Each UI's AuthorizationPolicy originally allowed requests only from cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account. Under a PERMISSIVE mesh that principal-based ALLOW matches nothing, so I replaced it with a path-based ALLOW.

The notebook-controller's ISTIO_GATEWAY env defaults to kubeflow/kubeflow-gateway, a stub Gateway in the kubeflow ns that isn't bound to any ingress IP in my setup. I override it to istio-system/kubeflow-gateway with a kustomize configMapGenerator using behavior:merge. The per-notebook VirtualService the controller emits then binds to the real ingress, and traffic to /notebook/<ns>/<name>/ routes correctly.

Ten surfaces read the user header: centraldashboard, kfam, manager, jupyter-web-app, katib-ui, kserve-models-web-app, volumes-web-app, tensorboards-web-app, ml-pipeline-ui and ml-pipeline-api-server. Most use USERID_HEADER; ml-pipeline-ui and ml-pipeline-api-server use KUBEFLOW_USERID_HEADER. Get it wrong on any single component and that UI answers "no user identity in header" on every request, even though oauth2-proxy has "authenticated" the user.

One more quirk. jupyter-web-app's /api/namespaces/<ns>/notebooks listing endpoint returns a 500 for any Notebook CR with a container that has no volumeMounts field. The web UI always adds a workspace volume to the notebooks it spawns, so you only see this with directly-applied CRs with bare containers. The bug is at utils.py:145, which does cntr["volumeMounts"] without a key check. Workaround: always add at least one volumeMount, even an emptyDir.

The Jupyter web app's notebook list in a Profile namespace, with a running GPU notebook (nvidia.com/gpu: 1) and its Connect button.

Two notebooks, one shared volume

A notebook's storage is only as useful as your ability to get at it from somewhere else. A teammate's notebook, a second kernel, a training job that needs the same dataset. So I tested the thing the Jupyter form quietly offers: a volume that more than one notebook can mount and write to at the same time.

The storage underneath is CephFS, through the ceph-filesystem storage class with ReadWriteMany. The default block class is ReadWriteOnce, which gives one pod its own disk and nobody else.

Create the first notebook with two shared volumes. In the "New notebook" form, untick "Use default class" on the workspace volume, pick ceph-filesystem and ReadWriteMany, then add one data volume the same way:

Ignore the "nfs" in the name. I named the notebook before I'd settled on the backend, and there is no NFS server in this cluster.

The "New notebook" form with the workspace volume set to ceph-filesystem and ReadWriteMany.

Check that both volumes are mounted, and that they really are two volumes. Open a terminal in JupyterLab:

df -hT /home/jovyan /home/jovyan/pod-nfs-shared-nb-shared 

Filesystem Type Size Used Avail Use% Mounted on csi-cephfs-node@<cluster-id>.ceph-filesystem=/volumes/csi/csi-vol-ebf4d59a-... ceph 5.0G 0 5.0G 0% /home/jovyan csi-cephfs-node@<cluster-id>.ceph-filesystem=/volumes/csi/csi-vol-198d5e31-... ceph 10G 0 10G 0% /home/jovyan/pod-nfs-shared-nb-shared 

Two different csi-vol IDs and two different sizes. It isn't one volume with a subfolder in it.

Write a marker file from the first notebook:

echo "written by $(hostname) at $(date)" > /home/jovyan/pod-nfs-shared-nb-shared/proof.txt cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt 

written by pod-nfs-shared-nb-0 at <timestamp> 

Create a second notebook and attach the same volume. I called it share-test. Under Data Volumes, choose "Attach existing volume" and pick pod-nfs-shared-nb-shared. The dropdown lists every claim in the namespace next to its access mode, which is a quiet but useful piece of UI: you can see which volumes are shareable before you try.

Read the file from the second notebook, then reply. Both notebooks are running:

cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt echo "reply from $(hostname)" >> /home/jovyan/pod-nfs-shared-nb-shared/proof.txt 

written by pod-nfs-shared-nb-0 at <timestamp> 

Back in the first notebook, read it again:

Source：https://dev.to/ezejioforog/brownfield-kubeflow-on-rke2-swapping-dex-seaweedfs-for-keycloak-ceph-rgw-4bdl
My favorites

Related posts

Three dates that catch a hard-coded time-zone bug in a session clock

A session that starts at 08:00 local time does not necessarily start at the same UTC hour all year. Less obviously, the

0dev.to

Laravel N+1 Query Problem: How to Find and Fix It

When a Laravel page that used to load in half a second suddenly takes four, the cause is very often the same: the N+1 qu

0dev.to

LongCat-2.5 at $0.30/$1.20: Thinking Locks Temperature, 1M Context, and Vendor Benches Between Opus 4.8 and 5

Attributed compile + one small offline run on my box Primary source: Introducing LongCat-2.5 — Native Multimodality for

0dev.to

Shipped: two packages, a registry listing, and the first seals

What happened Post 1 ended with a list: the verify package inside Fiador — human-backed attestation for Discord bots — i

0dev.to

AI Agent Memory Needs Garbage Collection, Not Just Storage

TL;DR — Most long-term memory systems for AI agents only know how to write and append — never to forget, consolidate, or

0dev.to

Securing the Analytics Pipeline and Pivoting for Demo Day

Sprint Retrospective: Implementing Zero-Trust Analytics The conclusion of Sprint 11 was entirely focused on access contr

0dev.to