Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

Self-Hosted Databricks Alternative: Homelab Data Lakehouse

Part of the homelab build series. Trino, Apache Iceberg, Project Nessie, dbt, Spark, MLflow
and OpenMetadata on RKE2, with highly available state on Ceph and CloudNativePG, secrets from
Vault and sign-in through Keycloak. Raw data comes in at one door, gets built and inspected
on the line, and leaves as tables and models I can trust. Nobody sends an invoice.

Introduction

A factory you rent by the hour, or one you own

Raw data is worth about as much as raw steel. Nobody wants it until someone has cut it,
joined it, checked it and stamped it.

Snowflake, Databricks and BigQuery will rent you an excellent factory for that, and the
meter runs for as long as the lights are on. I wanted the factory without the meter, so I
built one on three mini-PCs in my homelab, on the same RKE2 cluster as everything else in
this series. Every station is a real open-source tool doing a real job.

It's built to make one thing: machine-learning models trained on data I can name down to
the exact version. One is already serving, a football predictor that a scheduled pipeline
refits and re-tests every day.

Here is the floor, in the order the material moves:

• The loading dock: Ceph object storage. Everything that arrives lands here.
• The stock ledger: Project Nessie. Every table is an Apache Iceberg table, and Nessie records every version it has ever had. It's Git for data.
• The assembly line and quality control: dbt. Tables are built and tested on a branch, so a batch that fails inspection never reaches the shelf.
• The front counter: Trino. Anyone with access asks for what they need in plain SQL.
• The showroom: Superset. Charts and dashboards over the finished tables, for people who'd rather not write SQL. It asks for data in their name, never its own.
• The records office: OpenMetadata. It traces any table back to its raw material.
• The machine shop: Kubeflow. Notebooks, pipelines and hyperparameter searches, on GPUs when a job needs them.
• Finished goods: MLflow and KServe. Every run is tracked, and the model that passes its checks is served over HTTP.

The utilities were already in the building: Ceph, CloudNativePG, Vault, Keycloak, Istio and
ArgoCD.

It's highly available where it holds state. Postgres runs three instances, one in each
datacenter, the ledger runs two replicas and search runs three nodes. Trino, Superset,
OpenMetadata and MLflow run a single replica each, and I come back to that below. One bay is still
empty: a Hermes AI copilot that will read the lineage over OpenMetadata's MCP
interface, waiting on a DGX Spark that hasn't arrived.

If you know Databricks, you already know the layout:

It's an open-source equivalent, not Databricks, which is a product and a trademark, and
there's no affiliation. This is the second design: the first used Hive Metastore and
Airflow, and I say below why I replaced both.

The brief

The brief I set myself: an MLOps and big-data platform on the existing homelab RKE2 cluster
that a real data team could plausibly use. The homelab's existing backends (Rook-Ceph,
CloudNativePG, Vault, Keycloak) had to thread through every component, and each piece had
to land in GitOps and come back after a node reboot with nobody touching it.

What landed is two tiers. The lakehouse: Ceph RGW object storage, Apache Iceberg
tables, a Project Nessie catalog (state in CNPG nessie_db), Trino from Stackable's
coordinated 25.3.0 release, dbt-trino for tested marts, and OpenMetadata for
governance, with the Kubeflow spark-operator installed for the day the data outgrows
SQL. MLOps on top: MLflow Tracking and the Model
Registry, backed by CNPG mlflow_backend_db and the mlflow-artifacts bucket, behind
oauth2-proxy for the browser and reachable directly over in-cluster DNS from Kubeflow
notebooks.

The constraints that shaped every decision:

• No HDFS, no ZooKeeper. Ceph RGW is my S3, and nothing in the lakehouse needs ZK. The RAM saving alone justified leaving both out; the complexity avoided mattered more.
• No bundled databases. Every chart wants to ship its own MySQL or Postgres or Mongo. I ripped each one out of the lakehouse and MLflow charts and pointed them at CNPG. There is one Postgres cluster for this stack (postgres-mlops-cnpg-cluster, three instances across three DCs, each on its own zone-local volume) and it hosts the lakehouse and MLflow databases. (Kubeflow is the exception; it keeps its bundled MySQL, katib-mysql and Model Registry's Postgres for now.)
• No v1/Secret literals in Git. Every credential (application user passwords, OAuth2 client secrets, OIDC cookie keys) flows from Terraform into a Vault path, then into a VaultStaticSecret CR, then into the destination namespace. Kustomize may declare a placeholder Secret to keep references valid, but VSO always wins on the data values.
• Mirror production patterns, even when it's harder. Real OIDC, real mTLS, real RBAC. No "developer convenience" backdoors.

The rest of this article covers how each piece lands, what I cut, and which
traps cost the most time.

The architecture

If the word is new, the lakehouse is the third step in a short history. A warehouse
gives fast SQL over the tables you planned for, in a format you rent. A lake keeps any
file cheaply but has no table rules, so teams copy the data into a warehouse anyway and
end up with two versions of the truth. A lakehouse puts the table rules (Iceberg) and
the versions (Nessie) on the open files themselves, so every engine reads one copy.

Warehouse, lake, lakehouse: a warehouse sells storage and engine together, a lake plus a warehouse keeps two copies, and the lakehouse built here keeps one copy of open files in Ceph with Iceberg tables and Nessie versions read by Trino, dbt, Kubeflow and MLflow

Here is how this one is put together:

The data factory:** dbt, Trino, Nessie, Kubeflow, MLflow and KServe on the line, Superset and OpenMetadata above it behind Keycloak, and Ceph, CloudNativePG, Vault, Istio and Argo CD underneath

With five guided views: the query path, the daily pipeline, governance, train-and-serve,
and where secrets come from. Blog platforms can't embed the HTML, so host it and link it.

Kubeflow appears here as one station, the machine shop; how I changed it from upstream
is in the SSO section. Read the diagram bottom-up, because that's the order the
dependencies run:

• Foundation. Ceph RGW holds every byte of table data (s3://mlops-data-lake, TLS-only at s3.georgehomelab.com). One CloudNativePG cluster holds the lakehouse and MLflow stores (nessie_db, mlflow_backend_db, openmetadata_db). Vault + VSO hand out every credential.
• Catalog. Project Nessie answers "where does table X live, and at which commit?" over Iceberg REST, versioning the whole catalog like Git.
• Query and transform. Trino reads and writes the Iceberg tables; dbt builds and tests the marts through Trino every morning, on a Nessie branch dbt-audit that only merges into main if the tests pass.
• Govern. OpenMetadata 2.0.2 (search on its own OpenSearch 3.4.0) crawls Trino at 10:00 and ingests dbt's artifacts at 10:15. That's where the lineage comes from. People reach it through Keycloak (Google or GitHub).
• Consume. Superset 6.1 gives people charts, dashboards and a SQL editor over Trino. It signs people in through Keycloak and asks Trino to run each query as the person, so Trino's rules and its audit table see a name, never a shared login.
• Train and serve. MLflow tracks runs in Postgres and models in RGW; a KServe InferenceService serves the promoted model.

Everything in the foundation layer predates this work. Most of my effort went into
making sure the upper layers didn't reinvent any of it. The one deliberate exception is
OpenMetadata's own OpenSearch; the going-live section explains why that stopped being
optional.

Stackable operators, and what I dropped

Stackable ships a coordinated release of operators for Hadoop-ecosystem services. It's
tempting to take all of them. I only took the parts that fit:

The Stackable operators pin to 25.3.0 and release as a set, so I bump them as a set
(never Trino alone). Project Nessie and OpenMetadata are not Stackable products; they
deploy from their own Helm charts, the price of a best-of-breed catalog and governance layer.

The secret-operator needs explaining. The workload operators' admission webhooks
reference the SecretClass CRD, and without it workload CR creation fails. So I install
the operator but create no SecretClass of my own. The chart's default tls SecretClass
issues Trino's internal (autoTls) certificates, and the dbt and ingestion pods mount its
CA to verify them. Credentials never go through it; they're plain v1/Secrets synced by
VSO from Vault.

Stackable workloads also brought the first Istio trap. Under STRICT mTLS (the cluster
default), the API server's webhook calls can't reach the operator pods, because the API
server has no mesh client cert. The fix is a PeerAuthentication: PERMISSIVE in the
mlops namespace. Don't turn off injection; relax the mode for that namespace and
nothing else.

Storage: Rook-Ceph all the way down

The homelab already runs Rook-Ceph with four storage classes: ceph-block (RBD, RWO,
three copies, the default for notebook PVCs and most stateful pods), ceph-block-pg-local
(RBD, one copy, pinned to a datacenter, for Postgres only), ceph-filesystem (CephFS, RWX,
only when a workload really needs shared scratch) and ceph-bucket (RGW, S3), the data lake.

RGW is TLS-only. Clients use https://s3.georgehomelab.com,
which terminates on the Istio gateway with a public Let's Encrypt certificate (no private
CA to distribute), and path-style addressing still works. The old plain-HTTP
…rook-ceph.svc:80 endpoint is gone.

Buckets come from Rook's ObjectBucketClaim CR. The OBC controller provisions the bucket
on RGW and creates a per-bucket Secret with AWS_ACCESS_KEY_ID /
AWS_SECRET_ACCESS_KEY keys, named after the claim: Spark reads mlops-data-lake, MLflow
reads mlflow-artifacts, KFP reads kfp-artifacts. Nothing fans out from a single root key.

apiVersion: objectbucket.io/v1alpha1 kind: ObjectBucketClaim metadata: name: mlops-data-lake namespace: mlops spec: bucketName: mlops-data-lake storageClassName: ceph-bucket 

Apply that and the bucket, its owner user and the Secret all exist, ready for envFrom.

The Ceph dashboard → Object Gateway → Buckets: mlops-data-lake, mlflow-artifacts and kfp-artifacts, each with its own OBC-issued owner.

Databases: one CNPG cluster, many logical databases

The easy path is to leave every chart's postgresql.enabled: true on and accept the
extra StatefulSets. I didn't: backups, upgrades, monitoring and HA belong on one tier, not
nine. The homelab's CNPG pattern already runs Keycloak, SonarQube, Tekton and others, so
one more cluster meant copying an existing manifest (trimmed here; the real one adds
managed roles, affinity and resources):

apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: name: postgres-mlops-cnpg-cluster namespace: mlops spec: instances: 3 imageName: ghcr.io/cloudnative-pg/postgresql:17 bootstrap: initdb: database: mlops_placeholder # real DBs come from Database CRs owner: postgres secret: name: mlops-cnpg-bootstrap # VSO-synced from Vault storage: storageClass: ceph-block-pg-local # one Ceph copy, in the instance's own DC size: 40Gi postgresql: parameters: max_connections: "200" backup: retentionPolicy: "7d" barmanObjectStore: destinationPath: "s3://cnpg-backups/postgres-mlops-cnpg-cluster" # Ceph RGW 

Three instances, one per DC, automatic failover, and continuous WAL archival to RGW.

The storage class is worth a sentence. Postgres already keeps three copies of the data,
one per instance. On the default ceph-block, Ceph kept three copies of each of those,
nine in all, and every commit waited on a write to another datacenter. So Postgres now
sits on ceph-block-pg-local: one Ceph copy, in the same datacenter as the instance.
Postgres does the replicating and Ceph just holds the disk. A 4K synced write on a
database node went from about 34 ms to between 15 and 24 ms. The price is that losing a
drive loses that instance, and the operator has to clone a new one from the primary.
CNPG's Database CR (its per-logical-DB declaration) provisions one logical database per
app, each with its own owner role whose password lives at its own Vault path:

The lakehouse pivot added nessie_db and openmetadata_db with a short Database CR
each, and no new stateful tier. The old hive_metastore_db and airflow_metadata_db
(and their hive / airflow roles) are retained but no longer used: their Database
CRs set databaseReclaimPolicy: retain, so nothing drops them by accident.

Each app sees only its own DB, with its own creds. The superuser never lands on
an application Pod.

kubectl get database.postgresql.cnpg.io -n mlops: one Database per app on the single postgres-mlops-cnpg-cluster, all APPLIED true, including the retained (unused) hive_metastore_db and airflow_metadata_db.

Two Istio nuances are specific to CNPG. First, its istio-injected pods cycle their
sidecars whenever Istio config changes, so the READY column bounces between 1/2 and
2/2 and the operator reports READY < INSTANCES. Postgres is fine; it's sidecar churn.
Don't triage CNPG; bump the sidecar resources or ignore it.

Second, any istio-injected namespace must run at PSS=privileged, because istio-init
needs NET_ADMIN/NET_RAW. CNPG's initdb Job lives in such a namespace. If PSS is
baseline, the Job goes FailedCreate with no obvious cause. That's why the mlops
namespace is PSS=privileged.

Secrets: Vault all the way down

Vault is the single source of truth for credentials, and Terraform is the only thing that
writes to it. The chain for any credential: Terraform reads the value from a gitignored
.secrets.auto.tfvars and writes a Vault KV-v2 path; a VaultStaticSecret CR names
that path, the Vault role and the Secret to produce; VSO reconciles it hourly into a
plain v1/Secret (optionally reshaping keys with a template); the workload mounts it.

apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: name: mlops-mlflow-db namespace: mlops spec: vaultAuthRef: mlops-mlflow mount: secret type: kv-v2 path: homelab/mlops/mlflow-db refreshAfter: 1h destination: name: mlops-mlflow-db create: true overwrite: true 

Don't drop the overwrite: true. If the same Secret is referenced from both a kustomize
secretGenerator (with placeholder data) and a VSO sync, every kubectl apply -k briefly
stomps VSO's data with the kustomize placeholders. With overwrite: true on the VSS, VSO
puts the real values back on its next reconcile. Without it, oauth2-proxy gets
unauthorized_client from Keycloak and you spend two hours blaming the realm import.

Each app has its own ServiceAccount, its own Vault role (Kubernetes auth, policy scoped to
its own paths) and its own VSS, so no credential crosses an app boundary.

The Vault UI → secret/ → homelab/mlops/: the list of per-app paths (mlflow-db, nessie-db, openmetadata-db, …). ⚠️ Show the path list only; .

SSO: per-app oauth2-proxy with Keycloak upstream

Every UI in this stack sits behind Keycloak, using the standard
Authorization-Code-with-PKCE flow. For MLflow: the Istio gateway routes
mlflow.georgehomelab.com to its oauth2-proxy, which looks for the _oauth2_proxy_mlflow
cookie. Without it, the browser goes to
https://keycloak.georgehomelab.com/realms/mlflow/protocol/openid-connect/auth?..., signs in
with Google, and comes back to /oauth2/callback. oauth2-proxy validates the token, sets the
cookie, and from then on reverse-proxies to mlflow.mlops.svc.cluster.local:5000 with
X-Forwarded-Email added.

Each app gets its own realm too: MLflow's proxy uses realm mlflow, and OpenMetadata
(which speaks OIDC natively) uses realm openmetadata.

The decision that isn't obvious is running one oauth2-proxy per app rather than one
shared proxy. One shared proxy means one shared cookie. A leaked MLflow session becomes a
leaked session for OpenMetadata, Kiali, Hubble UI and every other gated app at once.
Per-app instances cost a little RAM (oauth2-proxy is ~30 MiB) and buy real session
isolation. Each overlay sets OAUTH2_PROXY_COOKIE_NAME (_oauth2_proxy_<app>) and
OAUTH2_PROXY_COOKIE_DOMAINS (its own host). The base values use a default that collides,
so without the override the first app to set a cookie "captures" it and later apps return

1. The per-app client secret follows the usual chain: Terraform writes Vault, a VaultStaticSecret syncs it into the namespace, and oauth2-proxy reads it via envFrom.

Two more lessons came out of this. In oauth2-proxy's proxy mode (which I use for
MLflow), the user email reaches the upstream as X-Forwarded-Email, not
X-Auth-Request-Email. The latter is a response header for auth_request-style
integrations. Get it wrong and the upstream app sees no user identity even though
authentication "worked."

And Keycloak doesn't substitute environment variables at realm import. The realm JSON
includes the oauth2-proxy client, but its secret is the literal placeholder string
${TOOL_OAUTH2_PROXY_CLIENT_SECRET}. Every new realm needs a post-import
kcadm update clients/<id> -s secret=<actual> patch with the same value that's in Vault.
post-deploy-fixes.sh automates it, but "Invalid client credentials" on a fresh realm
still catches you out the first time.

The Keycloak sign-in page for the mlflow realm, with the "Sign in with Google" and "GitHub" buttons. Every UI in this stack sends you to a page like this first.

Brownfield Kubeflow: the same pattern, applied to a stack that ships its own auth

Upstream Kubeflow ships its own login stack: Dex as the identity provider, with
oauth2-proxy acting as an Istio ext_authz check in front of it. I don't run Dex at all.

What changed, checked against the live cluster:

• Dex → Keycloak. Login goes Google → Keycloak realm kubeflow → oauth2-proxy (kubeflow-sso/oauth2-proxy-kubeflow, 2 replicas) in reverse-proxy mode, with one --upstream per app. The upstream per-app hosts: ['*'] VirtualServices are deleted.
• The user header. Every app reads X-Forwarded-Email instead of upstream's kubeflow-userid. That's nine separate places to patch, and all of them must agree.
• Authorization. The AuthorizationPolicies are path-based (paths: ["/*"]) instead of principal-based, because the mesh is PERMISSIVE. /notebook/* is gated by a CUSTOM ext_authz policy on the gateway.
• Istio. The cluster's shared Istio, with its own istio-system/kubeflow-gateway and a Let's Encrypt certificate.
• Storage. KFP artifacts go to an ObjectBucketClaim on Ceph RGW instead of the bundled SeaweedFS/MinIO, with the secret from Vault via VSO; every PVC is on ceph-block.
• GPUs. The NVIDIA gpu-operator runs three RTX A1000 cards, and notebooks claim one each.
• Still upstream. KFP and ML Metadata run on the bundled MySQL (a CNPG database exists for them but nothing uses it yet), Katib keeps katib-mysql, and Model Registry keeps its own Postgres. The bundled MySQL is at least backed up now: a daily CronJob dumps it to its own bucket on RGW.

The catalog: Project Nessie (Iceberg REST + Git-for-data)

The catalog is the small service everything else depends on. It remembers that "table events_curated
is an Iceberg table whose current metadata lives at s3://mlops-data-lake/warehouse/...,
with these columns, these snapshots, these partitions." Replacing the Hive Metastore with
Project Nessie was the main reason for the pivot.

Why Nessie over HMS:

• It's an Iceberg REST catalog, the modern standard. Trino, Spark and dbt all speak Iceberg REST; HMS's Thrift is the legacy path.
• Git-for-data. Nessie versions the entire catalog: branches, tags, commits, merges. You can cut a branch, run an ETL job, validate it, and merge atomically, which gives you multi-table transactions HMS can't do. On an ML platform that means reproducible data: tag the exact catalog state a model trained on. Both halves are in real use: dbt builds on a branch, and the football model trains on a tag (more on both below).
• Credential vending. Nessie holds the S3 creds and vends them to the engines when they load a table. (Without STS on Ceph, "vending" hands out the same bucket-owner key, and as the going-live section shows, vending alone isn't enough for Trino to create a table.)

Nessie deploys from its own Helm chart (charts.projectnessie.org), not Stackable. Its
version store is Postgres (CNPG nessie_db), and its warehouse is the same
mlops-data-lake bucket Spark writes to:

versionStoreType: JDBC2 jdbc: jdbcUrl: "jdbc:postgresql://postgres-mlops-cnpg-cluster-rw.mlops.svc:5432/nessie_db" secret: name: mlops-nessie-db username: username password: password catalog: enabled: true iceberg: defaultWarehouse: warehouse warehouses: - name: warehouse location: s3://mlops-data-lake/warehouse storage: s3: defaultOptions: endpoint: https://s3.georgehomelab.com pathStyleAccess: true accessKeySecret: name: mlops-data-lake awsAccessKeyId: AWS_ACCESS_KEY_ID awsSecretAccessKey: AWS_SECRET_ACCESS_KEY 

Trino reaches it at http://nessie.mlops.svc:19120/iceberg. Spark uses
org.apache.iceberg.nessie.NessieCatalog against .../api/v2 and can create and merge
branches from a job.

The Nessie UI on main: the commit log and the namespaces (raw, raw_football, marts_staging, marts_marts, marts_football) that dbt and Trino have created.

Why Nessie and not lakeFS? Nessie already gives the Iceberg tables Git-style
branches, tags and atomic multi-table commits; a second version-control layer over the
same tables would be two sources of truth for one question. lakeFS earns its place for
unstructured files (I have none yet), and its Iceberg REST catalog, SSO and RBAC are
paid-edition features. So I'll revisit it as a replacement, never an addition, if
Nessie stalls or unstructured data shows up.

Garbage collection. Versioning has a cost that's easy to miss: every dbt build and
pipeline publish replaces table snapshots, and the old files stay forever unless something
removes them. Nessie's separate GC tool runs as a weekly CronJob (Sundays, 13:00 UTC).

It marks the files still reachable (30 days of history per branch, and just the tagged
commit for each train-football-… tag) and treats the rest of each table's files as
candidates. It only looks under Iceberg table locations, so the landing Parquet is out of
its reach. It runs with --defer-deletes for now, recording candidates in nessie_db
until I've reviewed a few runs. The first walked 3 references and 131 commits and found 8
candidates, all old table metadata.

A daily tag-retention job sits in front of it, so the pins don't pile up either. It
keeps any tag a Model Registry version names, plus the last 30 days, and is also in
dry-run mode for now.

MLflow: tracking and model registry

MLflow is the simplest part of the stack to deploy and the one I use most day to day.
The chart is the official one from the MLflow repo, vendored at mlflow/charts/mlflow
(chart 0.1.0, appVersion 3.12.0). It takes a Postgres backend and an S3 artifact store.

values.yaml excerpt (trimmed):

mlflow: backendStoreUriFrom: secretKeyRef: name: mlops-mlflow-db # VSO builds the `uri` key from Vault key: uri defaultArtifactRoot: mlflow-artifacts:/ # clients use the proxy scheme artifactsDestination: s3://mlflow-artifacts/ # the server does the S3 I/O env: - name: MLFLOW_SERVER_ENABLE_JOB_EXECUTION value: "false" - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: mlflow-artifacts key: AWS_ACCESS_KEY_ID - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: mlflow-artifacts key: AWS_SECRET_ACCESS_KEY - name: MLFLOW_S3_ENDPOINT_URL value: https://s3.georgehomelab.com - name: MLFLOW_S3_IGNORE_TLS value: "false" 

Artifacts are proxied: clients log to mlflow-artifacts:/ and the server writes to the
bucket with its own OBC credentials, so notebooks only need the tracking URI, not S3 keys.
The MLFLOW_S3_ENDPOINT_URL override points the server at RGW instead of AWS. The
endpoint has a public certificate, so MLFLOW_S3_IGNORE_TLS stays false: TLS is
genuinely verified. Path-style addressing works because the host is in the
CephObjectStore's hosting.dnsNames.

MLFLOW_SERVER_ENABLE_JOB_EXECUTION=false is there because MLflow 3.x's job subsystem is
on by default and starts eight huey consumers, which ate about 1.5Gi while completely idle
and kept OOM-killing the pod. I don't use server-side jobs, so turning it off beat raising
the limit.

The M...