The first time an interviewer asked me how Iceberg handles a DELETE, I gave the copy-on-write versus merge-on-read answer and felt great about it. Then they asked what happens to read performance after a few hundred small deletes pile up on the same files. I had nothing. I knew the vocabulary and had never thought about the mechanics.
That was a while ago. The mechanics just changed. Between March and August 2026, Snowflake, Databricks, and Amazon Redshift all shipped support for Apache Iceberg v3. Upgrading an existing v2 table is a single property change, format-version set to 3. On most platforms that change is permanent, and every engine that touches the table has to keep up with it.
I think v3 is the most consequential table format change most Data Engineers will deal with this year. The upgrade is easy to run and hard to undo, and the questions it raises (delete handling, lineage, schema evolution) are concepts that transfer across every engine, which is why they're showing up in interview loops.
Deletion vectors replace position delete files
In v2, a row-level delete under merge-on-read wrote a position delete file: a separate file listing (data file, row position) pairs that readers had to join against at query time. Run enough small deletes and you get a pile of tiny delete files, and every read pays to reconcile them. Anyone who has run a busy merge-on-read table without aggressive compaction knows this pain. You run MERGE, it's fast, and 3 weeks later your dashboards are slow and nobody knows why.
v3 replaces that with deletion vectors. The Apache Iceberg spec says the current representation is "deletion vectors (v3 or above), which encode deleted positions of a single file in a bitmap," and that position delete files "may be created in v2 tables only." Writers can't add new position deletes to a v3 table, and each data file gets at most one deletion vector per snapshot.
The internals are worth knowing:
- The vector is a Roaring bitmap stored as a
deletion-vector-v1blob in a Puffin file. Bit P set means row P is deleted. - Each blob records the
referenced-data-fileand acardinality(the count of deleted rows). - The byte framing is big-endian, which the Puffin spec says was chosen for compatibility with Delta deletion vectors. The 2 big lakehouse formats are quietly converging on the same delete encoding (make of that what you will).
Now the performance claims. Databricks says deletion-vector DML is "up to 10x faster" than copy-on-write. Dremio reports a 50 to 80% read improvement over positional deletes from "our testing." Neither publishes a workload, dataset size, or method. AWS ran a real benchmark on EMR 7.10 with Spark 3.5.5 and Iceberg 1.9.2, and that's the one I'd read before believing any number. Use the vendor figures to justify a test, not a migration.
Here's the part I like best, because it's an economics question. One vector per data file means new deletes get merged into the existing vector. The cost moves rather than disappearing. You stop paying for a swarm of tiny delete files at read time, and you start rewriting one blob per data file at write time as deletes accumulate. For most tables that's a great trade, since reads outnumber writes by a mile. For a table taking constant small deletes on the same hot files, measure it. That's my inference from the spec, not a benchmark.
Migration is also lazy. Per Dremio, the first DELETE, UPDATE, or MERGE after upgrade converts existing position deletes into deletion vectors. Until then, engines keep reading the old v2 delete files. If a table never gets a row-level write, it keeps its v2 delete debt indefinitely.
Row lineage and the fine print for CDC pipelines
Every v3 row carries 2 new pseudo-columns:
-
_row_id: a unique long identifier for every row in the table. -
_last_updated_sequence_number: the sequence number of the commit that last touched that row.
The obvious use is incremental processing. Store a watermark, pull everything with a higher sequence number, done. AWS's Big Data Blog goes as far as saying "row lineage and deletion vectors make CDC a native property of the table itself."
I've been through enough "native CDC" pitches to read the fine print first. Here it is.
Deleted rows disappear. Lineage stamps surviving rows with the commit that last changed them, and a deleted row has no row left to stamp. From what I could read in the spec, nothing in lineage itself tells a consumer "this row is gone." You still need snapshot diffs or delete metadata to catch deletes. Redshift's docs list lineage as populated on INSERT, CTAS, UPDATE, and MERGE, and DELETE is missing from that list.
Equality deletes break lineage. The spec says lineage isn't tracked for rows updated through equality deletes, and a review comment on the lineage PR states that "when row lineage is enabled, new snapshots cannot include Equality Deletes." Equality deletes are the standard upsert path for Flink-style streaming writers. If your pipeline depends on them, you need a different write design, or that table stays on v2. (Streaming is overrated for most of y'all anyway, but if you're running it, this is your blocker.)
Updates may not keep their ID either. The spec lets engines model an update either as a modification that preserves the row ID or as a delete plus an insert. A consumer that assumes _row_id survives an UPDATE can break when a different engine writes the table.
The upgrade also leaves a gap. On Redshift, both columns return NULL for pre-upgrade rows until the first write, and that first write "generates row lineage values for the entire table." AWS also says the upgrade "does not perform a historical backfill of row-lineage change tracking records." Plan that first write as a deliberate baseline. Until it runs, your watermark query can't see a single old row:
SELECT _row_id, _last_updated_sequence_number, order_id, status FROM analytics.orders WHERE _last_updated_sequence_number > 1842; Those columns are excluded from SELECT *, so you have to name them. I've debugged a "the incremental job returned nothing" incident more than once, and it's always something this boring.
Column defaults and schema evolution after an in-place upgrade
v3 adds column defaults, and the spec defines 2 per field:
-
initial-defaultfills the value for every record written before the field was added. It's set only when the field is added. -
write-defaultcovers records written after. It starts equal to the initial default and can change through schema evolution.
The pitch is "add a column with a default and skip the rewrite." That's true for reads: old files return the default without being touched. New writes are different. Writers materialize the write default into new files, so changing it later only affects future rows, and old files keep whatever they had. Redshift's version of this: SET DEFAULT "applies to data files written after the change."
Then each engine adds its own wrinkles, and they don't agree:
- Snowflake:
ADD COLUMN ... DEFAULTsets both defaults. You can't change the initial default afterward; you drop and re-add the column.SET WRITE DEFAULTchanges only the write default. - Databricks: per its v3 docs, "Databricks supports write defaults. Databricks doesn't support initial defaults."
- Redshift: literal defaults only, no defaults on nested types, CTAS doesn't inherit defaults, and a default on a v2 table returns an error.
Put those together. Add a column with a default in Snowflake, then read the table from Databricks, and the pre-existing rows could plausibly come back NULL. I haven't run that test, and you shouldn't trust it from me or anyone else until you've run it on your own tables. Also, since the spec assigns a new field ID when a field is added, a wrong initial default fixed by drop-and-re-add probably leaves the old values orphaned under the old ID. That's an inference too. Test it.
One more blocker: the spec says a writer must fail on a required field with no non-null write default, and Snowflake's 2026_05 behavior bundle makes ADD COLUMN ... NOT NULL without a DEFAULT an error on v3 tables. If you have migration scripts that do that, they'll stop working.
The upgrade itself
On Spark with Iceberg, the upgrade is a single statement:
ALTER TABLE lake.analytics.orders SET TBLPROPERTIES ('format-version' = '3'); In Snowflake, active position delete files can block the upgrade, so you clear them first:
ALTER ICEBERG TABLE analytics.orders OPTIMIZE (REMOVE_POSITION_DELETES); Migrations are mostly compaction projects; that's been true of every warehouse migration I've done at 3am, and it's true here.
The question that matters is whether you can go back. Snowflake's upgrade docs say "the upgrade is permanent" and "all engines that read from or write to an upgraded table must support Iceberg v3." Your rollback there is Time Travel to the pre-upgrade state. Redshift doesn't support downgrades. AWS's S3 guide calls it "a one-way upgrade" and says that "Amazon Athena can't read V3 tables." Databricks documents a RESTORE-based path to a pre-upgrade version. So the format is one-way, and each platform's restore is a separate, platform-specific escape hatch.
The upgrade is one line of SQL. The blast radius is every reader you forgot you had.
I guarantee somebody in your org has a "temporary" Athena query feeding a finance spreadsheet. Upgrade the table and that report breaks, and you'll hear about it from the CFO's chief of staff.
Before you flip the switch, inventory 3 things:
- Every reader and writer. Spark versions, Trino, Athena, Teradata, that one Python script. Snowflake's GA release still says external-engine writes to Snowflake-managed v3 tables through Horizon Catalog aren't supported.
- Types. Redshift v3 rejects struct, list, map, variant, geometry, geography, binary, uuid, time, and nanosecond timestamps. Databricks supports VARIANT but not nanosecond timestamps. If your wide denormalized tables lean on STRUCT and ARRAY (and they should), Redshift is out for those tables today.
- Features, not just version. Compatibility depends on which features you enable.
initial-defaulton required fields breaks old readers. AWS warns that compaction writes shredded variant Parquet by default, which older readers might fail on (write.variant.shredding.enabled=falseavoids it). Redshift mapstimestamptzto TIMESTAMPTZ instead of TIMESTAMP, so your query output changes.
Why table format questions now show up in interview loops
I can't point you to a dataset of interview questions; nobody publishes those reliably. From both sides of the table, I've seen that once a technology ships on every major platform, it stops being "nice to know" and becomes a way to separate candidates who've run production tables from candidates who've read a blog post. Iceberg hit that point this year.
I'd also argue these are good questions, which I rarely say about interview questions. They test concepts. The same delete-file, compaction, and schema-evolution trade-offs apply to Delta and Hudi too. Bit manipulation never comes up on the job (I still couldn't solve one of those to save my life). "Why are reads slow on this merge-on-read table" comes up constantly.
If I were prepping, I'd be ready to answer these out loud, with trade-offs:
- How did v2 handle row-level deletes, and what does v3 change? Position delete files versus one Roaring bitmap per data file. Mention that the cost moves to write time.
- What happens to existing position deletes after upgrade? They stay valid and get converted lazily on the next row-level write.
- Can you use row lineage as a CDC feed? For inserts and updates, with a baseline write after upgrade. Deletes need another mechanism, and equality-delete upserts conflict with it.
- How would you add a NOT NULL column to a 10TB table without rewriting it? Initial and write defaults, plus the engine-specific gotchas.
- Walk me through upgrading a production table to v3. Inventory readers, check types, compact position deletes, plan rollback, plan the lineage baseline. If you say "it's just a metadata change," you've failed the question.
The hands-on round still exists too. Expect to write a MERGE, explain what it does to the files underneath, and reason about the small-files problem afterward. For that side of it, spark practice is covered on datadriven.io, and we built it around exactly the "write it, then explain what it did to storage" style of question.
The tools will change again in 18 months. Iceberg v4 will show up with its own one-line upgrade and its own list of broken readers. The concepts underneath (how deletes are represented, what a schema change costs, who reads your table) will still be the interview, and they'll still be the job.
So here's my question for the people who've already done this in production: what reader broke first when you set format-version to 3, and how long did it take someone to notice?
