All About Iceberg's File Format API: simplifying the engine/format interface and Parquet's multimodal future

What is the file format API?

Apache Iceberg traditionally exposed its data files through three libraries, one each for Parquet, Avro and ORC, and until this year every engine that spoke Iceberg had its own way of calling them. Spark had SparkParquetReaders, Flink had FlinkParquetReaders, the generic Java API had GenericParquetReaders, and each engine chose its own path through the format's read options, filter pushdown, split handling, encryption and metrics. The practical effect was that a feature added to one format for one engine (a new v3 type, a default value, a shredded variant column) tended to arrive for the other engines later or not at all, because the work had to be redone per engine.

The File Format API, which shipped in 1.11.0 and was announced in February, is the project's attempt to streamline all this: a single registry of "format models," each one the pairing of a file format with an engine's in-memory row type, and a single set of builders (ReadBuilder, FileWriterBuilder and its delete-file siblings) that every engine now goes through. It was also originally framed as the way to try out newer formats such as Vortex and Lance, which are fundamentally designed very differently than Parquet and meant specifically for vector work and multimodal data storage.

Note for Rust/Python users: this is a Java API. The registry lives in iceberg-core, so it reaches exactly the engines that link that jar and reuse its readers (Spark, Flink, Hive, the plain Java Record path). DuckDB, DataFusion via iceberg-rust, PyIceberg and Polars each implement the Iceberg spec with their own readers and leverage Arrow readers instead. For Trino, which is Java, it instead uses Iceberg's core for metadata and its own Parquet and ORC readers for data. DuckDB, DataFusion, Polars and PyIceberg all read into Arrow or something Arrow can hand over without copying, so the row-type axis collapses to one and a new format just needs an Arrow reader (Lance and Vortex both ship one). Java implementations have three row types and no shared memory format, which is why it needed a registry to play the role Arrow plays everywhere else.

Table 1. Which engines the File Format API actually reaches.

Reducing Format Model x Reader Complexity

A file format is how bytes are laid out on disk: Parquet, Avro and ORC each have their own encoding, compression, statistics and footer, and Iceberg doesn't care which one a given data file uses as long as it can read it. An engine in-memory row type, which the project calls the object model, is how one row looks in memory once it's been read, and that depends on the engine, not the file: Spark wants an InternalRow, Flink a RowData, the plain Java API an Iceberg Record. The same Parquet file can be read into any of them, and the same InternalRow can be written out as Parquet or ORC. A format model is one cell in that grid: one format, one row type.

Here is the grid as it stands at 1.12.0, with a mark wherever a format model exists:

Table 2. The format-by-row-type grid. Each "yes" is one registered format model.

Scale this complexity further, per engine and per format:

After, through the File Format API's registry:

**The API makes the engine's row type pluggable over a fixed list of file formats, such that any engine can register its own representation and read every Iceberg table through the shared builders, while a new format remains a change inside the Iceberg project; and because the formats are fixed, multimodal support is arriving as new column types defined on the Parquet side (see FILE type that was shipped in parquet-format 2.14.0, VECTOR type currently in proposal stage) and mirrored into the next table spec.

Into the weeds

Everything in the API is built on one map, which makes it the ideal place to start digging into in the codebase to gain a better understanding. If you only want to read and write files through the registry, skip ahead to "Usage." Read this section to understand why the API behaves the way it does, and come back to it when something in a later section needs the detail.

The registry key

A map here is the ordinary programming kind: a lookup table from a key to a value. In FormatModelRegistry, the key is a pair (which file format, which row type) and the value is a format model, the object that knows how to read and write that one combination. Here is the method that puts entries into the map:

// core/src/main/java/org/apache/iceberg/formats/FormatModelRegistry.java (1.12.0) public static synchronized void register(FormatModel<?, ?> formatModel) { Pair<FileFormat, Class<?>> key = Pair.of(formatModel.format(), formatModel.type()); ... MODELS.put(key, formatModel); } 

The second half of the key, Class<?>, is the Java class of the row the engine wants back. For the plain Java API it is Iceberg's own Record; for Spark it is InternalRow (one row at a time) or ColumnarBatch (a block of rows); for Flink it is RowData. The Iceberg project calls this the object model.

[!note] Object model versus schema
The object model is the engine's in-memory shape for a row. It says nothing about the file and nothing about the table's schema. The schema says which columns a table has; the object model says what Java class a row of that table becomes once it is read.

The first half of the key, FileFormat, is an enum: a type whose allowed values are a fixed list written into the source code. You cannot add a value at runtime. This enum lives in iceberg-api, the module that holds Iceberg's public interfaces:

// api/src/main/java/org/apache/iceberg/FileFormat.java (1.12.0) public enum FileFormat { PUFFIN("puffin", false), ORC("orc", true), PARQUET("parquet", true), AVRO("avro", true), METADATA("metadata.json", false); ... } 

Puffin and metadata appear because the enum names every kind of file Iceberg writes, not only data files. Puffin files hold statistics and deletion vectors; metadata.json is the table's root file. The three data formats are the ones in the middle: ORC, Parquet and Avro.

The two halves of the key are not the same kind of thing, and that difference organizes the rest of this post:

Table 3. The two halves of the registry key.

The announcement says the registry "decouples engines from specific formats and allows new formats to be added without modifying engine code." That is true as far as engine code goes. What a new format has to modify is iceberg-api, the enum itself.

What happens on a read

A scan task is Iceberg's description of one file to read, including which format it is in. From the task to the rows, a read goes like this:

1. The engine asks the registry for a read builder, passing the task's format, its own row type, and the file.
2. The registry looks up the (format, row type) pair.
3. If the pair is registered, the model returns a builder. The engine sets the columns it wants, a filter, and the byte range of the file it was assigned, then calls build().
4. If the pair is not registered, the registry throws an error that names the pair it could not find.

The dotted lines draw the open-versus-closed distinction: one input to the lookup comes from a fixed list, the other is whatever the engine says it is.

How models get registered

When the registry class is first loaded, it tries to call a register() method on four classes by name: the generic models, and the Arrow, Flink and Spark ones. If one of those classes, or something it depends on, is not on the classpath (the set of jars the program was started with), the registry logs a line and moves on. The practical consequence is that the contents of the map depend on which Iceberg jars you ship. You will see this directly in the "Usage" section. Everything the project registers at 1.12.0 is an engine's row type against a format Iceberg already had (Table 2); none of it is a new format.

Usage

It would be a terrible user experience to construct format models yourself; so instead you'll ask the registry for a builder and configure it. A builder is an object you set options on one at a time and then finish with build(). The registry exposes two of them, and together they are the whole surface an engine touches:

Table 4. The two registry calls an engine uses.

Where this code lives

If your experience of Iceberg so far is spark.sql("SELECT * FROM demo.orders") or a REST catalog behind a /v1/namespaces endpoint, the first question is where code like this even goes. It goes nowhere near either of those. There are three different places you can stand when you use Iceberg, and the File Format API only exists at the lowest one:

Table 5. Three levels of using Iceberg. The Format API is reachable only from the bottom one.

The program below is a plain Java main method with the Iceberg jars as dependencies. It is not a Spark job, it is not submitted to anything, and it does not need a catalog or even a table. It writes one Parquet file with Iceberg's own writer and reads it back with Iceberg's own reader, which is the single step every engine performs thousands of times per query and the only step the Format API is responsible for.

You would never run something like this in production. You run it when you want to see what the registry contains on a given classpath, or when you are writing a new format model and want the shortest possible loop between a change and a result. Here is the whole setup:

1. Make an empty directory with two files in it: pom.xml and src/main/java/demo/Main.java.
2. In pom.xml (Maven is Java's package manager, and this is its dependency list), declare the Iceberg modules you want on the classpath. The set is deliberately minimal: iceberg-api, iceberg-core, iceberg-data and iceberg-parquet, plus the Hadoop client jars Parquet needs and a logger. There is no iceberg-orc, and no Spark.
3. Run mvn -q compile exec:java. Maven downloads the jars from Maven Central, compiles Main.java, and runs it on the JVM in your terminal. The output below is what it prints.

The program

I ran this against Iceberg 1.12.0 from Maven Central on Java 17, with nothing timed. The program does four things:

1. Prints the FileFormat enum.
2. Reaches into the registry and prints what is in the map.
3. Writes three records through the write builder, then reads them back through the read builder.
4. Asks for a (format, row type) pair that cannot be there.

package demo; import java.io.File; import java.lang.reflect.Method; import java.util.Map; import org.apache.iceberg.DataFile; import org.apache.iceberg.FileFormat; import org.apache.iceberg.Files; import org.apache.iceberg.PartitionSpec; import org.apache.iceberg.Schema; import org.apache.iceberg.data.GenericRecord; import org.apache.iceberg.data.Record; import org.apache.iceberg.encryption.EncryptionUtil; import org.apache.iceberg.formats.FormatModelRegistry; import org.apache.iceberg.io.CloseableIterable; import org.apache.iceberg.io.DataWriter; import org.apache.iceberg.types.Types; public class Main { public static void main(String[] args) throws Exception { // 1. The format half of the key: a fixed list, nothing is added at runtime. System.out.println("== FileFormat enum values"); for (FileFormat f : FileFormat.values()) { System.out.println(" " + f + " splittable=" + f.isSplittable()); } // 2. Peek inside the registry (reflection, since the map is private). // Shows what the jars on the classpath registered before main() ran. System.out.println("== registered format models on this classpath (core + data + parquet)"); Method models = FormatModelRegistry.class.getDeclaredMethod("models"); models.setAccessible(true); Map<?, ?> m = (Map<?, ?>) models.invoke(null); m.keySet().stream().map(Object::toString).sorted().forEach(k -> System.out.println(" " + k)); // An Iceberg schema. Columns are matched by the numeric field id, not by name. Schema schema = new Schema( Types.NestedField.required(1, "id", Types.LongType.get()), Types.NestedField.optional(2, "name", Types.StringType.get())); // A local file to write to; an engine would get an object store path from FileIO. File out = new File(args.length > 0 ? args[0] : "/tmp/ffa-demo.parquet"); out.delete(); // 3a. Write. (PARQUET, Record.class) is the registry key; build() does the lookup // and hands back the Parquet model's writer, or throws if the pair is missing. System.out.println("== write via FormatModelRegistry.dataWriteBuilder(PARQUET, Record.class, ...)"); DataWriter<Record> writer = FormatModelRegistry.<Record, Object>dataWriteBuilder( FileFormat.PARQUET, Record.class, EncryptionUtil.plainAsEncryptedOutput(Files.localOutput(out))) .schema(schema) .spec(PartitionSpec.unpartitioned()) // required even without partitioning .overwrite() .build(); // Record is Iceberg's generic row type; copy(name, value, ...) makes a row. GenericRecord rec = GenericRecord.create(schema); for (long i = 1; i <= 3; i++) { writer.write(rec.copy("id", i, "name", "row-" + i)); } writer.close(); // toDataFile() returns the manifest entry an engine would commit, statistics included. DataFile df = writer.toDataFile(); System.out.println(" wrote " + df.path() + " format=" + df.format() + " records=" + df.recordCount() + " bytes=" + df.fileSizeInBytes() + " valueCounts=" + df.valueCounts()); // 3b. Read. Same key, other direction. An engine would also set a filter and a split. System.out.println("== read via FormatModelRegistry.readBuilder(PARQUET, Record.class, ...)"); try (CloseableIterable<Record> rows = FormatModelRegistry.<Record, Object>readBuilder( FileFormat.PARQUET, Record.class, Files.localInput(out)) .project(schema) .build()) { for (Record r : rows) { System.out.println(" " + r); } } // 4. A valid format and a valid row type, but iceberg-orc is not on the // classpath, so nobody registered the pair. The error names it. System.out.println("== ask for a model that is not registered (ORC with generic Record, no iceberg-orc on classpath)"); try { FormatModelRegistry.readBuilder(FileFormat.ORC, Record.class, Files.localInput(out)); } catch (IllegalArgumentException e) { System.out.println(" " + e.getMessage()); } } } 

And its output:

== FileFormat enum values PUFFIN splittable=false ORC splittable=true PARQUET splittable=true AVRO splittable=true METADATA splittable=false == registered format models on this classpath (core + data + parquet) (AVRO, class org.apache.iceberg.deletes.PositionDelete) (AVRO, interface org.apache.iceberg.data.Record) (PARQUET, class org.apache.iceberg.deletes.PositionDelete) (PARQUET, interface org.apache.iceberg.data.Record) == write via FormatModelRegistry.dataWriteBuilder(PARQUET, Record.class, ...) wrote .../out.parquet format=PARQUET records=3 bytes=684 valueCounts={1=3, 2=3} == read via FormatModelRegistry.readBuilder(PARQUET, Record.class, ...) Record(1, row-1) Record(2, row-2) Record(3, row-3) == ask for a model that is not registered (ORC with generic Record, no iceberg-orc on classpath) Format model is not registered for format ORC and type interface org.apache.iceberg.data.Record 

Reading the output top to bottom:

• The map holds Parquet and Avro, each paired with the generic Record and with the position-delete row type, and nothing else.
• Avro is present without being asked for, because its model ships inside iceberg-core.
• ORC is absent because the iceberg-orc jar is not on the classpath, and the error names the exact pair the registry could not find.
• The write returned valueCounts: the writer filled in the DataFile statistics without the program doing anything.
• One thing the program cannot express at all is a request for a format outside the five names. There is no FileFormat.LANCE to pass, and converting the string "lance" to a FileFormat throws "Invalid file format: lance" before the registry is ever reached.

The row-type side is open. The format side is closed.

The same call, from Spark

To connect this back to the level you already know: when you run spark.read.table("demo.orders").show() with the Iceberg connector, Spark loads the table through the catalog, plans which data files each task should scan, and then, for every file, makes exactly the call the program made by hand. Here is the Spark connector's version at 1.12.0, from BaseRowReader in the iceberg-spark module:

// Same call as the program above, with Spark's row type and all the options set. ReadBuilder<InternalRow, ?> reader = FormatModelRegistry.readBuilder(format, InternalRow.class, file); return reader .project(projection) // columns the query asked for .idToConstant(idToConstant) // partition values, not stored in the file .reuseContainers() .split(start, length) // this task's byte range .caseSensitive(caseSensitive()) .filter(residual) // what is left of the WHERE clause .withNameMapping(nameMapping()) .build(); 

The format variable is the FileFormat recorded in the manifest entry for that file, and InternalRow.class is the row type Spark registered for itself in SparkFormatModels at startup. The query you type at the top level becomes, at the bottom level, one registry lookup per file. Everything between the two is planning.

Exploring pluggability

The extension point is relatively easy to grokk and doesn't take a ton of code. This section covers what an engine supplies, what the format model owns, and what "pluggable" therefore means in practice.

Here is how the generic Parquet model is registered, with each argument labeled:

// data/src/main/java/org/apache/iceberg/data/GenericFormatModels.java (1.12.0) FormatModelRegistry.register( ParquetFormatModel.create( Record.class, // the row type Schema.class, // the engine schema type (icebergSchema, fileSchema, engineSchema) -> // how to write a Record GenericParquetWriter.create(icebergSchema, fileSchema), (icebergSchema, fileSchema, engineSchema, idToConstant) -> // how to read one GenericParquetReaders.buildReader(icebergSchema, fileSchema, idToConstant), new RecordVariantShreddingAnalyzer(), // see below (Function<Schema, UnaryOperator<Record>>) ignored -> Record::copy)); // how to copy a row 

The engine supplies two functions. The writer function, given a schema, builds something that writes a row's values into Parquet. The reader function builds something that reads values out of Parquet into a row; its idToConst...