Original Japanese article: Apache Iceberg v4の仕様を読んで、何が変わるのか考えてみた
Introduction
I'm Aki, an AWS Community Builder (@jitepengin).
Apache Iceberg v3 introduced deletion vectors and the Variant type, and services such as Amazon S3 Tables on AWS and Snowflake have been adding support for them.
At the same time, discussions about the next version, v4, have become quite concrete. In this article, I read the v4 spec and related proposals and thought about what will change, from a practitioner's point of view.
To give the conclusion up front: I think v4 is less about adding new features and more about the spec taking on work that we have been handling through operations until now.
In this article, I focus on the following three changes that caught my attention.
- The manifest list goes away and is replaced by a root manifest
- Data files and deletion vectors are stored in the same entry
- Writing new equality deletes is no longer allowed
References
- Iceberg Table Spec
- Iceberg V4 Adaptive Metadata Tree (proposal document)
- Iceberg Single File Commits (proposal document)
v4 Is Not a Finalized Spec Yet
The official spec page states that v4 is under development and has not yet been formally adopted.
That said, not everything is still at the proposal stage.
| Item | Status (as of November 2026) |
|---|---|
| Relative paths | Adopted by vote in May 2026 and included in the spec |
| Content stats (columnar statistics) | Adopted |
| Root manifest / adaptive metadata tree | Described in the spec, but discussion and implementation are ongoing |
| Ban on writing equality deletes | Listed in the spec's v4 changes |
Individual items being adopted is a separate matter from v4 as a whole being formally adopted.
Note: What follows is my own interpretation after reading the current spec and proposals. The final spec may change.
The Manifest List Goes Away and Is Replaced by a Root Manifest
In v4, the per-snapshot manifest list goes away, and each snapshot has a single root manifest instead.
The root manifest itself works as a manifest: it can hold data files directly and can also reference leaf manifests.
flowchart LR subgraph v3["Up to v3"] M3[metadata.json] --> ML[manifest list] ML --> DM["manifest file (data manifest)"] ML --> DLM["manifest file (delete manifest)"] DM --> DF3[data file] DLM --> DV3["deletion vector (Puffin)"] DV3 -. target .-> DF3 end subgraph v4["v4 (as described in the spec)"] M4[metadata.json] --> RM[root manifest] RM --> LM[leaf manifest] RM --> E1["data file entry (includes DV reference)"] LM --> E2["data file entry (includes DV reference)"] E1 --> DF4a[data file] E1 --> DV4a["deletion vector (Puffin)"] E2 --> DF4b[data file] E2 --> DV4b["deletion vector (Puffin)"] end Up to v3, even adding a single small data file requires writing a new metadata.json and a new manifest list. A manifest that records the added data file is also created, so metadata writes tend to be large relative to the amount of data.
If you commit every few minutes in a streaming workload, metadata grows faster than data.
In v4, small changes can be recorded directly in the root manifest as data files.
When the root manifest holds data files directly, there is no longer a need to prepare a separate manifest list and data manifest as before.
On AWS, Kinesis Data Streams can now deliver data directly to S3 Tables, and I think small, frequent commits will only become more common.
Until now, the growth of small data files has been handled by compaction, through S3 Tables' automatic maintenance or AWS Glue Data Catalog table optimization. Manifests also had to be reorganized with operations such as rewrite_manifests.
I think v4 aims to reduce the write overhead by rethinking the metadata structure at commit time itself, separately from these operational measures.
Also, in the spec, both the root manifest and leaf manifests are stored in Parquet. Today I inspect Avro manifests through PyIceberg, but once they are Parquet, it may become possible to query them directly from tools like DuckDB.
On the other hand, small changes can be recorded directly in the root, and a layout that uses leaf manifests is also possible when data files increase or when the metadata structure is reorganized.
As a result, the shape of the metadata changes depending on the state of the table and how it is written. The approach of reading metadata directly for investigation will probably need some relearning.
Delete Information Moves from Separate Management to Being Managed with the Data File
In v4, a data file and its corresponding deletion vector are stored in the same entry. The spec explains that this removes the need for two-phase planning.
However, the deletion vector itself is not embedded in the data file. The DV is still stored in a Puffin file, and its reference information is placed in the data file's manifest entry. What changes is not where the DV itself is stored, but where its reference information is managed.
In v2, position deletes and equality deletes were managed as separate files, and v3 added deletion vectors.
However, even in v3, DV reference information is managed in delete manifests, separate from the data files. In v4, placing that reference information in the data file's manifest entry makes it possible to handle data and delete information together.
That said, this does not make compaction unnecessary.
A deletion vector is a marker for "rows not to read," and the rows themselves remain in the Parquet file.
When physical deletion is required, as with deleting personal information, you need to rewrite files so they no longer contain the target rows, and also properly remove old snapshots and unneeded files.
Equality Deletes Can No Longer Be Written
This is the change that I think will have the biggest impact in practice.
The spec states that writing new equality deletes is not allowed in v4, and that equality delete files may be created only in v2 and v3 tables.
An equality delete expresses a deletion by value, such as "delete the row where id = 5." The writer can delete rows without knowing where they are, so equality deletes have been used in streaming and CDC. Implementations that support upserts, such as Flink's Iceberg sink, may use equality deletes. In exchange, readers must match the rows in data files against the conditions in delete files, which raises the cost of reads and compaction.
In other words, until now the structure was: "the write side takes the easy path, and the read side and operations pay for it." v4 moves that cost back to the write side.
From here on, this is my own interpretation, but I think this is a decision to avoid adding new equality delete processing to the read side.
In a lakehouse, there may be only a few engines that write, but the engines that read keep growing: Spark, Athena, Snowflake, DuckDB.
Rather than requiring every reading engine to handle new equality deletes, having the writing engines take on converting to deletion vectors or rewriting data files seems more efficient for the ecosystem as a whole. That is how I read it.
Compatibility for reading existing equality deletes is still required.
My understanding is that this change reduces dependence on equality deletes in future tables by banning only new writes.
In addition, in v4 identifier fields are replaced by a primary key constraint.
When upgrading from v2/v3, identifier fields are rewritten as a primary key constraint that is not enforced (it is not created if the fields include types that cannot be used, such as geometry or geography). I find it interesting that the removal of equality deletes and turning keys into constraints arrive in the same version.
This does not mean existing CDC pipelines will stop working right away.
Even in a table upgraded to v4, equality deletes written in the v2/v3 era continue to be applied. Only writing new ones is prohibited.
However, whether a table can be upgraded to v4 will likely depend on "whether every engine and tool writing to that table can update it without using equality deletes."
Other Changes That Caught My Attention
| Change | What the spec says | My thoughts |
|---|---|---|
| Relative paths | Relative paths can be used for paths in metadata | If files under the table are recorded with relative paths, it becomes easier to move a table without rewriting metadata. Bucket or Region migrations and DR may become easier |
| Table constraints | Table constraints are supported | Interesting when viewed together with the removal of equality deletes |
| File type | A type that holds bytes inline or as a reference to an external file | It may allow images, audio, and similar files to be handled as table data |
| Write default expressions | Value expressions can be used for write defaults | Default values become more expressive |
The one that caught my attention the most is relative paths.
The base directory mismatch between S3 Tables and Snowflake CLD that I wrote about before was also rooted in a gap between "the absolute path written in the metadata" and "the actual location."
A Base Directory Mismatch Between Amazon S3 Tables and Snowflake CLD
In v4, the location in table metadata itself can be omitted, in which case the side loading the table provides the location. It seems that whatever handles the catalog and table loading will take on more responsibility for "where this table is."
Conclusion
In this article, I read the Apache Iceberg v4 spec and thought about what will change.
I feel that the changes around manifests are especially significant.
- v4 has not been formally adopted yet, but some parts, such as relative paths and content stats, are already in the spec
- The manifest list is replaced by the root manifest, reducing the metadata overhead of small commits
- Deletion vectors are referenced from the same entry as the data file, but the DV itself stays in Puffin
- New equality deletes can no longer be written, and the cost of deletes moves back from the read side to the write side
- With relative paths, responsibility for a table's location shifts toward the catalog and whatever loads the table
All of the v4 changes seem to share a direction: reducing, by revisiting the spec, the work that operations and engines have absorbed until now.
On the other hand, I felt that in many cases, such as physical deletion in operations, CDC pipeline design, and catalog responsibility, the work does not disappear; it only changes where it is handled.
It will be a while before v4 is available, but now may be a good time to review "which processing is handled where in my own environment."
I hope this article is helpful to anyone following the Apache Iceberg v4 spec.



