Hosting an Entire Open Lakehouse on Port 8080

Most of my job, at least the part people see, is running Spark code in front of other people and promising it'll work. The catch is that I almost never demo the stable stuff. Nobody wants a talk about the version of Spark they're already running; they want the release that came out last month, or the feature that's still a JIRA ticket, so the thing I'm showing is usually newer than its documentation and occasionally newer than any release. Iceberg ships on its own schedule, both of them depend on an exact Hadoop and an exact AWS SDK, and the whole arrangement can quietly stop working between the week I build a demo and the week I give it without a single line of my code changing.

So in December 2025 I started lakehouse-at-home: somewhere real to install new versions of Spark and Iceberg and find out what broke before an audience did. This is how it got built, one service and one problem at a time. Each section adds to the architecture diagram; new pieces are highlighted, and everything built before them fades into the background.

Why on my own machine

Honestly, part of it was curiosity about the hardware. My desktop is a Ryzen 7 7800X3D with 64GB of RAM and an RTX 5070 Ti, which is a ridiculous amount of machine to spend mostly on a browser, and I wanted to know whether it could carry a real lakehouse: not single-node Spark reading a Parquet file, but the actual set of services a data platform runs on, all at once, talking to each other the way they would in production.

The rest was control. Managed platforms give you the versions they support, not the one that shipped last week, and certainly not a build from Spark's master branch, which is where I ended up more than once. A demo also has to behave the same way every single time; while preparing a recording in March I wrote the requirement down more bluntly than any design doc would: "keep in mind with demos theres a certain amount of cleanup and restarting needed for each session and predictable behaviour". Tearing everything down and bringing it back identical is trivial when the whole lakehouse lives on a box I own, and it costs nothing, which matters when you do it several times a day.

And I had a longer game in mind. If I was going to run a lakehouse at home anyway, it could eventually hold data that actually lives at home: readings from my IoT devices, my own documents, the kind of personal data I'd rather not ship to someone else's cloud to query. Local wasn't only the cheap option; it was the only one where the data could eventually be mine.

Starting from the data

I started from the tables rather than the compute, because the tables were the point. I wanted Iceberg and Delta side by side, reading and writing the same kinds of data, so that when a new release changed how one of them behaved I'd see the difference rather than read about it. (That paid off. Delta 4.0.1, for example, turned out not to load on Spark 4.1 at all, which I only found out because I tried.)

An open table format is really three things pretending to be one: an engine that reads and writes, a catalog that records which tables exist and which snapshot of each is current, and storage for the data files themselves. Each needs somewhere to live.

The simplest catalog that behaves like a real one is Iceberg's JDBC catalog, which keeps table metadata in an ordinary relational database, so I used Postgres, running natively on the host. For storage I wanted S3 semantics rather than a local directory, because every real lakehouse sits on object storage, so I picked SeaweedFS, which speaks the S3 API, and ran it natively too. On the Spark side, wiring that up is a handful of lines:

spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions spark.sql.catalog.iceberg org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.iceberg.type jdbc spark.sql.catalog.iceberg.uri jdbc:postgresql://localhost:5432/iceberg_catalog spark.sql.catalog.iceberg.warehouse s3a://lakehouse/warehouse 

And that's where it got hard, because Spark doesn't talk to S3 by itself. It goes through Hadoop's s3a connector, which is compiled against one specific version of the AWS SDK, and Iceberg's runtime has to agree with both. Get any of them slightly wrong and nothing tells you which one; you get a ClassNotFoundException, or a NoSuchMethodError from somewhere deep in the SDK, or a credentials failure that looks exactly like a credentials failure and isn't. Getting Spark to authenticate against SeaweedFS at all took a lot of trial and error with the s3a settings:

spark.hadoop.fs.s3a.endpoint http://localhost:8333 spark.hadoop.fs.s3a.path.style.access true # SeaweedFS isn't AWS: no virtual-host bucket URLs spark.hadoop.fs.s3a.connection.ssl.enabled false # plain HTTP on localhost spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem 

and then a lot more trial and error with the JARs, until the combination that worked was pinned so hard that the first README warns about it in the troubleshooting section, parenthetical and all: Hadoop 3.4.1 → AWS SDK v2.24.6 (not newer!). The download script from that week still pulls both AWS SDK bundles, the v1 bundle and the v2 one, because by that point I wasn't going to argue with anything that worked. All told it came to "~860MB of dependencies (exact versions required for compatibility)":

Most of what goes wrong when you upgrade a lakehouse isn't the engine; it's the dependency graph around it, and the only way to learn that graph is to break it somewhere safe. The project's agent guide still has a section called Critical Versions: Do not change without testing, which is the same lesson written for machines.

Once it worked, it was an ordinary lakehouse, and the first thing any test of it does is the ordinary thing:

CREATE NAMESPACE IF NOT EXISTS iceberg.bronze; CREATE TABLE iceberg.bronze.orders ( order_id STRING, customer_id STRING, amount DOUBLE, created_at TIMESTAMP ) USING iceberg; 

The last decision in that first week was where everything should run, and it shaped the next nine months more than anything else. Spark went in Docker, because I'd want to swap its versions constantly. Postgres and SeaweedFS stayed native on the host, because they were storage, and storage should outlive the experiments. The first compose file still shows the argument I had with myself about how the two sides would reach each other: a bridge network, published ports and a host.docker.internal gateway, all commented out, and above them the line that won:

spark-master: image: apache/spark:4.0.1-scala2.13-java17-python3-r-ubuntu network_mode: "host" # ports: # - "8090:8080" # Spark Master Web UI # - "7077:7077" # Spark Master # networks: # - lakehouse # extra_hosts: # - "host.docker.internal:host-gateway" 

Host networking made everything simple on my machine, and it's also the reason the title of this post is a port number. Every container shared the host's ports with every native service, so from then on each new component had to find a port nobody else was using. Keep an eye on the port numbers in the diagrams; they're the plot.

A catalog I had to build

Once I had tables, I wanted the kind of catalog people actually run in production: one that governs access, vends credentials, and lets more than one engine find the same tables. That meant Unity Catalog OSS, running alongside the Postgres catalog rather than replacing it, so both paths stayed testable.

The problem was that the features I wanted were ahead of the published images. When I was preparing an episode of OverArchitected in March, the release I needed didn't have a Docker image yet, so I cloned the repository, built the image myself, and ran a 0.5.0 snapshot from source. It was the first time the stack ran something that hadn't been released, and it wasn't the last. It also needed a port: its default was 8080, which Spark's UI already had, so it moved to 8081. From Spark's side it's just a second catalog, talking Iceberg's REST protocol:

spark.sql.catalog.unity org.apache.iceberg.spark.SparkCatalog spark.sql.catalog.unity.catalog-impl org.apache.iceberg.rest.RESTCatalog spark.sql.catalog.unity.uri http://localhost:8081/api/2.1/unity-catalog/iceberg 

Streams: Kafka and Zookeeper

Batch tables only tell half the story, so the next piece was streaming: Kafka for the events, with Zookeeper to coordinate it.

This part was harder than it had any right to be. When I finally sat down with it in January, the prompt was the whole mood of the week: "debug my kafka setup, why can't spark read it?" The answer was about eight small things at once. The streaming job never ran from_json() on Kafka's value column, so it was trying to aggregate raw bytes; a column was referenced as total.revenue instead of total_revenue; one alias was the wrong case; a debug query was blocking the stream. Kafka itself wasn't on host networking, so Spark couldn't reach it, and once it was, its connection string to Zookeeper was wrong. And Zookeeper's admin server, which listens on 8080 by default, was fighting Spark's web UI for the same port. Zookeeper lost, and moved to 9999.

The fix for the first of those is the part every Kafka tutorial skips: Kafka hands Spark bytes, and you have to tell it what they mean.

kafka_df = ( spark.readStream .format("kafka") .option("kafka.bootstrap.servers", "localhost:9092") .option("subscribe", "orders") .option("startingOffsets", "latest") .load() ) parsed = kafka_df.select( f.from_json(f.col("value").cast("string"), event_schema).alias("event"), f.col("timestamp").alias("kafka_timestamp"), ) 

Weeks later I was still citing Kafka as the cautionary tale; when I worried I was adding things too fast, the reason I gave was "how much trouble even setting up and testing kafka was".

Data that misbehaves

A lakehouse with no data in it proves nothing, and a lakehouse full of clean data proves less than you'd think: every pipeline looks correct when nothing is ever late, duplicated or missing. I needed data that behaved like the real thing, which is to say badly, and at a volume that would actually exercise the stack.

Rather than invent a domain, I borrowed one. Databricks' Casper's Kitchens simulates a ghost-kitchen delivery business, and I asked the agent to study how its generators worked and rebuild the idea for this stack ("please consult me first", I added, because I wanted to understand the shape of the data before it existed). What came out is a small Python package that simulates the full life of a food order:

Each step takes a Gaussian amount of time, demand follows the day, and some brands are growing while others decline, so there are real trends for a pipeline to find:

@dataclass class ServiceTimes: """Gaussian distribution parameters for order lifecycle phases (in minutes).""" order_to_kitchen_start: tuple[float, float] = (2.0, 0.5) kitchen_prep: tuple[float, float] = (15.0, 5.0) kitchen_to_driver: tuple[float, float] = (5.0, 2.0) pickup_to_delivery: tuple[float, float] = (20.0, 8.0) driver_ping_interval_sec: int = 60 

Demand peaks at noon and again from six to eight in the evening, and Fridays and Saturdays run 25 to 35 percent busier than a weekday. And then, on purpose, it breaks things:

@dataclass class ChaosConfig: """Data quality issue injection settings.""" enabled: bool = True null_rate: float = 0.05 # 5% of fields nullified  late_event_rate: float = 0.03 # 3% out-of-order events  duplicate_rate: float = 0.02 # 2% duplicate events  malformed_json_rate: float = 0.01 # 1% malformed body 

The same generator feeds both halves of the stack:

./lakehouse testdata generate --days 7 # history as Parquet ./lakehouse testdata load # into Iceberg bronze tables ./lakehouse testdata stream --speed 60 # replay into Kafka, an hour per minute ./lakehouse testdata stats 

The duplicates and late events are there because the interesting failures only show up when the data misbehaves, and months later, when I tested AUTO CDC, cancellations and duplicate creation events were exactly the cases the feature existed for.

Spark, twice

The point of all of this was testing Spark versions, and that's the part I designed most deliberately. Instead of upgrading 4.0 to 4.1 in place, the stack runs them side by side: each version gets its own compose file, its own container names, its own image (with the Java version the official image ships), and its own block of ports, so both clusters can be up at once and the same job can be submitted to either.

./lakehouse start spark # Spark 4.0.1 ./lakehouse start spark --version 4.1 # Spark 4.1.0, alongside it 

That turned out to be the organising idea for a lot of what followed. Spark Declarative Pipelines arrived in 4.1, so the workshop I built on the stack became a pair of notebooks: the same medallion pipeline written imperatively on 4.0 and declaratively on 4.1, running against the same Iceberg tables at the same time.

Orchestration: Airflow

Pipelines that only run when I type spark-submit aren't really pipelines, so in January the stack got Apache Airflow. The first version was Airflow 2.8, and within a week I upgraded it to 3.1, which was educational in the way major versions are: the webserver became an api-server, schedule_interval became schedule, BashOperator moved into a providers package, and the health endpoint moved.

In the middle of that upgrade, the Java version in one image flipped from 17 to 21 and back again, and my reaction at the time was correct, if not patient: "why do we keep switching java versions? this is a fundamental change we can't keep expecting users to keep up with". The rule that came out of it is that Java versions are whatever the official images ship, and nobody changes them.

The more useful correction was conceptual. The first architecture diagrams drew Airflow talking to Iceberg, and I asked the question that should have been asked earlier: "does airflow being linked to the iceberg api really make sense in the architecture diagrams? what is airflow really orchestrating in ths stack? just spark pipelines right?" It was. Airflow doesn't touch the data; it tells Spark when to, and the diagram got simpler once it said so:

The stack ships three DAGs:

The branch in the first DAG is my favourite detail in the repo, because the version-testing idea reaches all the way into orchestration:

choose_spark = BranchPythonOperator( task_id="choose_spark_version", python_callable=choose_spark_version, ) run_pipeline_spark41 = BashOperator( task_id="run_pipeline_spark41", bash_command=""" docker exec spark-master-41 /opt/spark/bin/spark-submit \  /scripts/pipelines/pipeline_spark41.py """, ) check_kafka >> choose_spark >> [run_pipeline_spark41, run_pipeline_spark40] [run_pipeline_spark41, run_pipeline_spark40] >> verify_tables 

Running declarative pipelines from Airflow was clumsier at first, because there was no operator for them; that changed in March, and I'll come back to it once SDP itself has been introduced.

One CLI, built for agents

By now there were two Spark clusters, a streaming stack, two catalogs, an orchestrator and a pile of native services, and remembering which compose file, which ports and which container names went with which version had become a job in itself. So everything went behind a single entry point, ./lakehouse, a bash CLI that wraps Docker Compose and hides the bookkeeping:

./lakehouse setup # prerequisites, Postgres catalog, JARs ./lakehouse preflight # do .env, Spark config and the catalog agree? ./lakehouse start all --version 4.1 # Spark + Kafka, then wait until they answer ./lakehouse start airflow # optional services, one at a time ./lakehouse status --json # machine-readable, for scripts and agents ./lakehouse logs spark-master ./lakehouse stop all 

Run it with no arguments and it explains itself, including one line that would matter later:

$ ./lakehouse help Lakehouse Management CLI Usage: lakehouse <command> [options] [--version 4.0|4.1] Commands: setup Validate environment and install dependencies check-config Validate credentials between .env and spark-defaults.conf preflight Run preflight checks before starting services migrate [--dry-run] Run database schema migrations status Show all services status start [service] Start service (spark|kafka|airflow|unity-catalog|all) ... testdata <cmd> Test data generation (generate|stream|load|stats|clean) browsedata [table] Browse Iceberg table rows (default: iceberg.bronze.orders) Services: ... all Spark + Kafka (not Unity Catalog or Airflow) 

all means the core services, which I discovered the hard way on the morning of a recording, after a reboot, when ./lakehouse start all brought back Spark and Kafka and not the catalog or MLflow I was about to demo.

The deeper reason for one CLI was that I wanted the stack to be agent-first. Early on I wrote down the goal: "I want people to feed an agent the repo, and be able to get up and running locally or on a cloud provider like AWS with minimal hitches." At the time, coding agents were far more reliable driving a well-behaved command line than reconstructing a sequence of docker compose invocations from a README, so the CLI was designed for them as much as for people. Commands fail with real exit codes and say what to do next, and status --json exists so an agent can read the state of the stack instead of scraping coloured terminal output.

Some features came straight from me asking the obvious question, like "is there a cli test process in setup that checks if ports are available for the services?"; there wasn't, so now start checks every port it's about to need and refuses to run if something else is holding one:

check_port_available() { local port=$1 local expected_container=$2 if ! nc -z localhost "$port" 2>/dev/null; then return 0 # port is free fi # port is in use: fine only if it's the container we expect if [ -n "$expected_container" ] && \ docker ps --format '{{.Names}}' | grep -q "^${expected_container}$"; then return 0 fi return 1 # blocked by something else } 

The documenta...