New this week: a condensed newsletter for data lakehouse and AI. Starting now, there is a shorter companion email that sums up the week in both the data lakehouse and AI worlds in one quick read. It is built for readers who want the headlines from both topics without two long issues. This longer newsletter is not going anywhere. It will keep coming every week with the same depth. To get the new condensed newsletter, subscribe at developer.dremio.com, where you can also download free books on Apache Iceberg, Apache Polaris, and the open lakehouse.

Three words sum up this week across the six projects: ship, settle, and specify. Polaris shipped 1.9.0 after a license problem forced a second release candidate. Arrow pushed out a wave of Rust, Go, and Java releases. DataFusion voted on three releases and on taking in a distributed execution library as a subproject. Then the spec work. Iceberg opened a formal vote to add materialized views to the view spec, a feature the community has debated for years. Parquet argued over page sizes, encryption, and what a statistic like distinct_count really promises. Ossie kept working on the hard questions of how a semantic model should behave when real engines read it.

A second thread ran under all of it. Several projects talked openly about how AI tools fit into open source work. Polaris moved toward an Assisted-by: line in commit messages. Iceberg C++ and Arrow debated whether automatic Copilot reviews help or just add noise. Ossie looked at an opt-in AI review bot. These are small process threads, but they show communities writing down their norms before the tools decide for them.

Apache Iceberg

Materialized views go to a vote

The biggest moment on the Iceberg list this week came on November 16. Jan Kaul called a vote to add materialized views to the Iceberg view spec, based on PR #11041. The vote followed a request for broader review that Jan sent on November 10 on behalf of the materialized view working group.

The design is simple to explain. A materialized view is an Iceberg view whose version carries a pointer to a storage table. The storage table is a normal Iceberg table that holds the precomputed results. Readers that do not understand materialized views ignore the pointer and run the view query directly. That choice matters. It means existing view consumers keep working with no changes, and engines can adopt the feature on their own schedule.

Refresh state lives in the snapshot summary of the storage table. It records which view version was materialized, when the refresh started, and the state of every source at refresh time. Tables are tracked by snapshot ID and views by version ID. With that record, any engine can decide whether the stored results are still fresh without asking the engine that wrote them.

Support on the vote thread was wide. Nevin Zheng, Marius Grama, Walaa Eldin Moustafa, Andrei Tserakhau, Benny Chow, Talat Uyarer, John Zhuge, Alex Stephen, Aastha Agrrawal, Pachu Shrinivas, and Huang-Hsiang Cheng all posted +1 votes. Alex Cruise voted +1 and raised two late questions. Walaa asked to keep the vote thread for votes and move the discussion to a new thread or the biweekly sync. In the earlier review thread, Walaa and Manish Malhotra thanked the group that carried the spec to this point, including Benny, Steven, Marius, Dan, Talat, Nevin, and Manish.

Why does this matter for lakehouse teams? Materialized views have lived inside individual engines for a long time, each with its own format and refresh logic. A shared spec means one engine can build a materialized view and another can read it and check its freshness. That turns a vendor feature into a property of the table format.

V4 takes shape, one argument at a time

A cluster of threads this week all point at the next format version. The most pointed one asked whether V4 should stop writing ORC and Avro data files. Maximilian Michels called it a tough one and then said he was +1 on dropping write support for both. His reasoning was blunt. Avro and ORC are not first-class citizens today, and the community has clearly chosen Parquet. He was firm that readers must keep reading both formats for backward compatibility.

Péter Váry pushed back on a single-format future. He argued that Iceberg needs at least two live, active columnar formats to keep innovation and composability alive. His picks were Parquet for stability and Vortex for newer work, while noting the community should decide. Zheng Hu proposed a different angle: give the file format API feature flags, so a table rejects features its file format does not support. That approach decouples table format evolution from what each file format can do. Renjie Liu and Ankit Kumar also joined in.

Huaxin Gao started a deep thread on cross-engine determinism for functions in stored expressions. The problem is real. Some V4 features compute a value at write time and compute it again later, possibly in a different engine. Check constraints, stats-only fields in the index spec, and derived columns all fall into that bucket. If two engines disagree on the result of a function, the table breaks in quiet ways.

Péter proposed replacing the current boolean deterministic flag with a volatility enum based on PostgreSQL's categories: VOLATILE, STABLE, and IMMUTABLE. He proposed deprecating the old flag with a defined mapping during a transition. Andrei Tserakhau backed the enum and added a key rule: what matters is whether a value gets computed again later, not the function on its own. Check validation, virtual columns, and stats-only expressions need IMMUTABLE. A value computed once and stored as the source of truth can be VOLATILE. Alex Cruise, Matt Butrovich, and Szehon Ho also weighed in.

On the vector type, Russell Spitzer laid out the options for float16 and int8 elements. The V4 vector(E, D) type stores fixed-length numeric arrays like embeddings, with a guarantee that every non-null value has exactly D finite, non-null elements. The proposed element types are the existing int, long, float, and double. Requests for float16 and int8 keep coming. Andrei voted to leave them out of V4 and add them as separate element types in V5 if needed. He pushed back on treating float16 as a storage choice for vector(float, D), since that forces every reader to support every encoding for a small size gain. Péter added a data point: with Parquet byte-stream split encoding, his benchmark showed float16 about 10% smaller. He asked the group to judge the options from the view of AI and ML workloads, since those drive the vector type in the first place.

Daniel Weeks created a rest-v4-changes branch to collect REST catalog spec changes for V4 tables, with the first changes in PR 17642. The branch holds only OpenAPI spec edits, which keeps rebases small. Jean-Baptiste Onofré promised a review.

Neelesh Salian also opened a thread on making format version 3 the default, following the same process the community used for the V2 switch. Java and Go already support most of V3. PyIceberg cannot write V3 yet, and iceberg-rust is still working through it. Gang Wu added that iceberg-cpp supports most V3 features except the variant type. Neelesh pointed to work in iceberg-verification to add V3 reference tables so implementations can confirm they read V3 the same way.

Indexes, UDFs, and the cost of equality deletes

The index work kept moving. Huaxin Gao posted notes from the index sync. The group agreed that index work should not wait on the table format timeline. Derived columns, such as storing Hilbert values, will move forward as a separate effort. Péter removed derived columns from the index spec PR (#16961) for now and posted the meeting recording on YouTube. Zheng Hu also responded to a long proposal on pluggable Puffin blob types and a compaction-aware lifecycle for file-scoped indexes, asking for a short summary to anchor discussion.

Yufei Gu proposed a biweekly UDF sync because UDF progress has been slow, partly from limited review capacity. Huaxin replied that he has the full UDF implementation locally and is landing it in small PRs. The first adds UDF leaf types (Representation and Parameter) in PR 15994. The list and load function REST endpoints are merged in PR 15180, and create, drop, and exists are in review in PR 17966. Ruobing Wang, Aastha Agrrawal, and Ajantha Bhat signed up to help with reviews.

The Kafka Connect CDC thread got sharper. Andrei summarized the proposal: the sink's coordinator converts each commit's equality deletes into deletion vectors before committing, so the sink never commits an equality delete. CDC is opt-in, and append-only stays the default. koodin9 posted benchmarks on PR 18003. With increasing IDs, a conversion stays under 0.4 seconds. With random UUID keys on a 10,000-file table, every file gets read, and conversion takes 2 to 42 seconds depending on storage latency. Maximilian confirmed the goal is to stop writing equality deletes entirely, but doubted that ad-hoc scanning scales. Grant Nicholas raised commit contention: writing deletion vectors instead of equality deletes creates contention with other writers like compaction, which hurts low-latency CDC workloads.

This is one of the most practical debates on the list. Equality deletes are cheap to write and expensive to read. Deletion vectors flip that trade. CDC pipelines sit right on the fault line, and the community is now measuring where the costs land.

Releases, Spark versions, and process

Maximilian Michels created a 1.12.x branch pointing at the 1.12.0 release tag, since important bug fixes have landed and a patch release looks likely. Anurag Mantripragada asked for two Spark fixes in a 1.12.1: one for concurrent runtime filtering on shared scans in copy-on-write UPDATE with a subquery (PR 18005), and one to stop reapplying stream-from-timestamp on stream resume (PR 18348). Yuya Ebihara flagged a failing CVE scan on the branch and opened PR 18945 to fix it.

Over in Go, Badal Singh proposed release blockers for iceberg-go v0.8.0. Each is serious: wrong results, rows coming back after deletion, or tables left unreadable. The lead issue, #2604, shows a merge-on-read delete on a table upgraded from V2 to V3 bringing back rows deleted before the upgrade. The new deletion vector does not include the older position deletes, and readers skip position deletes once a DV exists. Andrei proposed a narrow bar for blockers: regressions from 0.7.0, plus anything that silently loses data on a normal path. Upgrade paths are where format bugs hide, and this thread is a good reminder to test them.

Spark support keeps getting harder to manage. Anurag revived the Spark versioning discussion after Spark 4.3.0 shipped and Cheng Pan opened PR 18749 to add it. Anurag leaned toward skipping 4.3, a non-LTS minor with a six-month window, with Spark 4.4 expected around January. Cheng countered that the community already agreed on a policy of the latest LTS plus the two latest minors. Anurag then posted a proof of concept for a shared-source Spark layout in PR 19025. It moves 545 common source files from the 4.1 and 4.2 modules into spark/common, keeps full copies of any file that differs, uses no shims, and still compiles each minor against its own Spark. It also adds 4.3.

Several contributors also asked for reviews this week. Alex Prosak proposed initial snapshot load for Spark streaming reads in PR 18068, matching the TABLE_SCAN_THEN_INCREMENTAL strategy Flink has offered since 0.14.0. Today, a fresh Spark streaming checkpoint has to replay history from the oldest ancestor snapshot, which is slow and fails once snapshots expire. Sebastian Daberdaku asked for review of a fix that uses the catalog's Hadoop configuration when listing orphan files. Daksh Dave listed 12 open PRs across Core, Hive, AWS, and Spark. Hossein Torabi asked for a final review of the cache policy PR. Roman Denysov followed up on rounding before bucket hashing for nanosecond timestamps, with a spec clarification PR and a matching iceberg-verification PR.

The deepest spec-level debate of the week came from Sylwester Lachiewicz on name mapping for data files that carry only some field IDs. Manu Zhang said he is fixing the same inconsistency in iceberg-cpp. JB confirmed the Java behavior: the Parquet and ORC paths treat a file as having IDs if any field anywhere has one, so a file with only top-level IDs never uses the name mapping. Avro already works per field. He noted that a Java fix needs more than removing the hasIds() check, because the Parquet and ORC visitors overwrite existing IDs from the mapping. Gang Wu argued for keeping this implementation-specific, since files produced outside Iceberg can be broken in endless ways and the spec cannot chase every case. Ryan Blue joined the thread too.

On process, Manu Zhang proposed disabling automatic Copilot reviews in iceberg-cpp. His reasons: reviews at the default level are often low quality with false positives, and once quota runs out Copilot posts an "unable to review" comment on every push. Junwang Zhao and Gang Wu agreed. The iceberg-rust community also discussed its sync time after daylight saving time shifted it an hour earlier for US and European contributors. Renjie Liu and Marco Kroll supported pinning it to 8 AM Pacific year-round.

Finally, Danica Fine announced that the Iceberg Summit 2027 Call for Papers is open. The summit runs May 19 and 20 in Santa Clara, California, as two in-person days. This year it opens space for the broader open lakehouse ecosystem beyond Iceberg itself. Submissions close January 10, 2027, at 11:59 PM Pacific.

Apache Polaris

Polaris 1.9.0 ships after a second candidate

Polaris was the busiest list of the week with 150 messages. The headline: Apache Polaris 1.9.0 is out, announced by Jean-Baptiste Onofré.

The road there took two candidates. Alex Dutra cast a -1 on rc0 because the LICENSE file in the Spark bundle JARs was wrong. JSpecify was bundled but not listed, likely pulled in through the Iceberg 1.12 Spark runtime JAR. Checkerframework and JetBrains annotations were listed but not bundled. JB agreed, cancelled the vote, and did a full pass. The rc1 vote drew 18 messages and passed. Youngrae Kim's verification is a good model for anyone new to release voting: he checked every link, ran the verify script against the git SHA, confirmed all 468 signatures, checked the source tarball for binaries, and then started the server with in-memory persistence and ran catalog and table operations.

The release has fixes that matter in production. Table and view commits no longer delete newly written metadata when the metastore write outcome is unknown, such as a dropped connection during commit, while still cleaning up on known failures. That closes a nasty failure mode where a commit that actually succeeded loses its own metadata. Polaris now fully supports external principals, globally or per realm, through polaris.authentication.credential-mode=external paired with an external identity provider and an external policy decision point like OPA or Ranger. S3 storage configurations also gain an optional credentialVendingMechanism field, with STS as the default and only option for now. Xiaodong Deng noticed the apache-polaris Python package for 1.9.0 had not yet appeared on PyPI.

The license issue sparked a broader question about how the Spark plugin bundles polaris-core. Today the plugin includes polaris-core with transitive dependencies turned off, so the bundle holds core classes but none of their runtime dependencies. Yufei Gu pointed out that the Spark client uses only two classes from core, GenericTableEndpoints and PolarisResourcePaths. He favored moving them into a small polaris-api-catalog-common module with explicit dependencies, so Spark no longer depends on core at all. Dmitri Bourlatchkov, Ayush Saxena, and JB joined in.

JB then proposed Polaris 1.10.0 by mid-December and volunteered as release manager. He also proposed trying ATR, the Apache Trusted Releases platform built by the ASF Tooling team to modernize how projects build, verify, vote on, and publish releases. Robert Stupp agreed that the project should move toward ATR. He noted that SBOM work for Maven artifacts is close for 1.10, but reproducible SBOMs for the Python parts are still needed. He also said polaris-tools releases should not block ATR for main releases. Yufei raised a related point in a thread on distribution SBOMs: some dependencies inside the bundled Iceberg JAR do not show up in the SBOM, and he warned that consumers then read a missing entry as proof a component is absent.

Security, credentials, and authorization

Much of Polaris's work this week sat at the boundary between the catalog and cloud security. The remote signing thread restarted around a decision cache. Yufei concluded that invalidating a decision cache is hard, even with native RBAC, and that OPA and Ranger cannot support it either. He favored a TTL cache, with the maximum revocation delay written into the security contract. He pointed to AWS Verified Permissions, which caches authorization decisions for 120 seconds by default. Alex Dutra agreed the cache is reasonable if documented. He warned that his measured 19% overhead came from a laptop where Postgres calls are cheap, and he expects the cost to grow on Kubernetes in the cloud. In that case, he said, the cache becomes essential.

Yufei opened a thread on making storageName easier to use. Today, an admin creating a connection must know a name configured on the server, which is awkward when catalog admins and deployment admins are different people. Dmitri tied it to the older secrets manager discussion and suggested reviving it, since Polaris has only an in-memory implementation today. In a related thread, Prithvi S explained the case for static AWS keys for SigV4 catalog auth: Iceberg REST catalogs that speak SigV4 but have no IAM role or STS endpoint. Think on-prem catalogs or anything behind a SigV4 front door that only accepts a static key pair. Today those catalogs cannot be federated. Arnab Karmakar proposed a custom STS endpoint for federated catalog connections, and Dmitri linked the implementation in PR 6035.

The credential vending discussion branched across clouds. Vignesh A and Yufei went back and forth on vending against non-public Azure and GCS endpoints for private-link setups. Yufei endorsed a plan for named base storage credentials for Azure and GCS. Dmitri posted threads on Hierarchical Namespace support for GCS credential vending and Cloudflare R2 support with scoped credentials. Serhii Svynarchuk raised a bug around custom metadata paths for views: creating or updating a view on S3 failed with HTTP 403 when write.metadata.path pointed outside the view location, because Polaris left that path out of the scoped credentials. Ayush agreed the scoping fix is clearly correct and asked whether the more lenient compatibility behavior for views, compared with tables, is deliberate.

On authorization, JB described a use case from a Polaris users meeting in France: external principals with Polaris-managed principal roles. With external credential mode, a principal's roles come straight from the token and have no grants in the metastore, so the mode requires an external authorizer. JB proposed a middle ground. Alex Dutra supported it, noting it is close to the long-discussed "federated" mode but persists only roles ahead of time and keeps principals external. Laurent Goujon asked whether his SPI PR for chain-aware view authorization, built on Iceberg's referenced-by feature, can leave draft. Yufei said yes. Dmitri merged the Ranger follow-up for the new authorization check on GET /v1/config and proposed enabling ENFORCE_CATALOG_CONFIG_AUTHORIZATION by default in internal tests as a first rollout step. Vignesh and Yufei also agreed on which client-facing keys belong in /v1/config defaults. Adnan Hemani raised four new lineage privileges from the OpenLineage integration PR: read and ingest lineage for tables and for views. Yufei gave a +1.

Persistence, APIs, and operations

Ayush Saxena flagged a new relational JDBC schema version so operators hear about it before the release notes. PR 5089 adds a table that tracks realms whose time-series data is still pending deletion. Today, purging a realm leaves its events and metrics reports behind with nothing to reclaim them. JB supported the approach and pointed out two other schema PRs in flight, including PR 6013, which bumps the schema to v7 with a new index on events. He noted that CREATE INDEX IF NOT EXISTS keeps old index definitions on existing deployments, so changed indexes need explicit handling.

Srinivas Rishindra asked for explicit sign-off on pagination rules for Polaris-native list APIs in PR 6028, because the rules differ from the Iceberg REST spec. A client must always check for a page token. JB agreed: Iceberg promises that a request with no page token gets the full result, but new Polaris-native operations make no such promise, so "always follow the token" is the cleaner contract and bounds response size from the start.

The soft-delete proposal drew careful review. Dmitri was wary of expiring soft-deleted entities during REST requests against other entities, calling it too magical for users. He suggested the Admin Tool as the right home for expiry. Alex Dutra added a sharp detail from his remote signing work: Spark's DROP TABLE ... PURGE does not ask the catalog to purge. Spark calls drop with purge set to false and deletes files itself through the table's FileIO. On a catalog with soft-delete, that means a purge request puts the table on hold with its name reserved. Engine behavior shapes catalog design more than spec documents suggest.

Arnab also asked for input on an admin report for tables incompatible with structured locations. Dmitri suggested the name check-entity-locations and recommended reading the structured setting from RealmConfig. Yufei said the metrics REST spec is ready to merge, thanking Anand for the work. JB and a Kafka extension contributor discussed how the Iceberg REST event endpoint and Polaris event payloads relate. JB explained that the Iceberg endpoint uses a schema owned by the Iceberg spec, a pull API with continuation tokens, so it will not return the Kafka payload directly. Youngrae Kim proposed making uri optional for Hadoop connection configs, since the Hadoop federated catalog never uses it and valid warehouse URIs are rejected today. Alexandre Dutra posted a proposal for asynchronous and reliable tasks. Yufei opened a thread on Load Table v2 and what it means for Polaris. Alex Dutra raised an Apache Cassandra backend for NoSQL persistence.

Two process threads round out Polaris. Alex Dutra revived the AI-generated code disclosure discussion and proposed adopting Dmitri's suggested Assisted-by: <Name of Tool> <Version> line. The plan puts the requirement in AGENTS.md, keeps CONTRIBUTING.md asking for disclosure in PR descriptions, and adds a PR checklist item. Yufei gave a +1. And the changelog.d PR merged, moving Polaris to changelog fragments to cut merge conflicts on CHANGELOG.md.

Apache Arrow

A release-heavy week

Arrow's list was mostly release votes, and that is a sign of a healthy project. Jeffrey Vo ran three Rust patch releases in parallel: 57.3.3, 58.4.2, and 59.3.3. All three passed with votes from L. C. Hsieh, Xuanwo, Raúl Cumplido, and others. Maintaining three older major lines at once shows how many downstream projects pin to specific arrow-rs majors. Jeffrey also announced the 60.2.0 plan and opened the 60.2.0 RC1 vote.

Matt Topol ran the Arrow Go 18.9.0 vote, which passed with five binding +1 votes from Bryce Mecum, David Li, Raúl Cumplido, Gang Wu, and Matt himself.

JB proposed Arrow Java 20.0.0 RC0 on November 17. Binding +1 votes came in quickly from Raúl Cumplido, Sutou Kouhei, Jeffrey Vo, Andy Grove, Gang Wu, Bryce Mecum, and David Li. Andy Grove verified it by upgrading Comet locally, which is exactly the kind of downstream test that catches real problems. Hongye Wan, a new contributor, checked the signatures and checksums and was clear about what he did not test.

Fixing the Java release rhythm

JB used the 20.0.0 vote to propose a regular Arrow Java release cadence. His diagnosis was honest. The 20.0.0 release took longer than planned because the team kept pulling in incoming PRs. Since the release cycle has been inconsistent, contributors pushed to get changes into each release. He proposed releasing Arrow Java at least quarterly, or more often on demand, and volunteered to drive it. Antoine Pitrou replied that a predictable cadence has helped every project where he has seen it, while noting he is not a Java contributor. Andy Grove also joined.

This is a small thread with a big lesson. Irregular releases create pressure to cram features in, which makes releases slower, which adds more pressure. A fixed cadence breaks the loop.

Copilot billing surprises and API questions

Jonathan Keane revived the Copilot review thread with a surprise. Automatic Copilot reviews on apache/arrow appear to count against his personal Copilot credits, even though he turned off automatic review requests in his own settings. Nic Crane explained that the copilot_code_review block in .asf.yaml overrides user settings and bills the user. He suggested limiting reviews to the initial PR instead of every push. Rok Mihevc also joined. It is a reminder that repo-level AI tooling settings have costs that land on individual contributors.

Samani Gikandi raised a design question about nested sort columns in arrow-go in PR 1337. He hit a snag keeping it backward compatible and asked two questions: what is the preferred compatible path, and what is the project's appetite for a breaking change. Hongye Wan introduced himself and his work on arrow-rs issue #11942, where cast_to_variant turns nested Variant storage into ordinary objects. Ian Cook announced the biweekly community meeting for November 18 at 17:00 UTC.

Apache Parquet

A new committer and a spirited week

The Parquet PMC welcomed Russell Spitzer as a committer, announced by Julien Le Dem. The thread collected 26 messages of congratulations, the busiest thread on the list. Russell already works across Iceberg and Parquet, and this week showed why that matters. He started one of the most important Parquet threads of the year.

Breaking DataPageV2 encryption, on purpose

Russell's thread opened with a hook: "I want to Break DataPageV2 Encryption in Parquet-Java". The problem is real. When modular encryption arrived, it treated DataPageV1 as one blob. Levels and values lived together, and encryption covered the whole thing, the same way compression did. DataPageV2 is different. Levels are not compressed with values. A V2 page holds uncompressed levels followed by compressed values. The parquet-java writer reused existing patterns and ended up writing unencrypted levels next to encrypted values. That breaks compatibility with every other Apache implementation.

Russell's position: fix it, even if it breaks older parquet-java readers. Julien Le Dem recalled that applying compression only to values in V2 was intended, since levels encoded with the RLE hybrid do not compress well and per-page compression stays optional. He agreed that encryption should cover the whole page and that this looks like an artifact of how encryption was implemented. Gidon Gershinsky, Antoine Pitrou, Prateek Gaur, and Xinli Shang all joined the 10-message thread. For anyone relying on Parquet modular encryption with V2 pages in Java, this thread is required reading.

A separate proposal from Akil Raza looked at extending Parquet Modular Encryption beyond AES to value-level protection and externally supplied cryptography. Adam Reeve asked whether finer-grained keys already cover the use case, and noted that envelope encryption with a custom KMS client already lets organizations plug in their own service.

Page sizes, statistics, and encodings

Alkis Evlogimenos proposed removing the 8KB page size recommendation from the configuration guide. His case was clear. Pages compress independently, so small pages limit compression. They also inflate page indexes: for the same data, 8KB pages mean roughly 128 times as many entries as 1MB pages. parquet-java, Arrow C++, and Arrow Rust all target 1MB pages by default. Andrew Pilloud agreed and added that most blob stores are tuned for reads of 1MB or more. At 8KB, reading and decoding page headers likely costs more than reading the data. Jigao Luo, Prateek Gaur, Russell Spitzer, and Andrew Lamb also weighed in. This is a docs change, but docs drive defaults in new writers, so it matters.

The distinct_count thread turned into a debate on intent. Adrian Garcia Badaracco proposed classifying the field as inexact. Daniel Weeks was fine with a clarification but objected to redefining it as an estimate. He felt the argument leaned on the field never being clearly specified, rather than on what it was introduced to do. Adrian asked for evidence that anyone ever understood the field as exact. Query engines use this statistic for planning, so the answer has teeth.

Encoding work moved forward too. Gunnar Morling reported that Hardwood now supports ALP encoding for floating-point data and interoperates bit-exactly with parquet-java's implementation in both directions. He described one difference: Hardwood builds each column chunk's preset from eight sample vectors spread across the whole chunk. Gang Wu, Antoine Pitrou, and Prateek Gaur also took part. Andrew Lamb responded to Prateek's FastLanes benchmark breakdown and recommended naming the proposal FASTLANES, since the term already has mindshare. Prateek worried the term is overloaded but said he was fine with it.

New types and new layouts

Neelesh Salian split the logical type out of the earlier WIRE proposal into a standalone SERIALIZED_VALUE logical type for serialized messages, following Micah's suggestion. Projection will come later as a separate proposal once one or more engines agree to use it. Matt Topol pushed back. He still does not see the use case, since these messages are usually intermediate data passed between systems, not something written raw to Parquet for storage. He suggested a struct column for the readers and writers that care. Julien Le Dem and Scott Haines also joined.

The FILE logical type kept moving. Gunnar asked what is left to merge fixtures for the FILE type, and Andrew Lamb promised to review it. In a related thread on ambiguity when inline and uri are both set, Daniel Weeks said inline values are meant to match the referenced part of the external file, and a mismatch is an error state the spec should not try to define. Stefan Grafberger summarized the agreement: writing both is useful, inline is a copy of the referenced data, and readers are not required to recheck the external data, though they can with a checksum.

Thomas Kissinger posted objections on the extensible decimal floating-point type, focused on the BID encoding. He called it overloaded and non-trivial, with almost no ecosystem support across languages. Divjot Arora shared a page-first data layout proposal that builds on the earlier non-contiguous pages idea and uses the modular footer. His core point: the row group and column chunk structure works when columns are uniform in size but breaks down when sizes or densities vary. Jiayi Wang posted updates to the modular footer Thrift definitions and asked for help auditing which existing metadata fields to keep, remove, or redesign.

On governance, Julien said he addressed feedback on the Parquet versioning taxonomy of changes in parquet-site PR 220 and announced the Parquet sync for November 18. Manu Zhang announced Parquet Java 1.18.2, a maintenance release with fixes for column-path handling in Bloom filters and dictionary lookups, better writer failure handling and row-group buffer cleanup, protobuf bytes fixes, and dependency updates.

Apache DataFusion

Three releases and a new subproject

DataFusion ran a lot of releases this week. Andrew Lamb ran votes for 54.2.0 and 55.3.0. Both passed. Andrew reported that 54.2.0 passed with six +1 votes, three of them binding, and posted it to crates.io. He also sent the 55.3.0 result. Starry Hao, Martin Grigorov, L. C. Hsieh, Marko Milenković, and Adrian Garcia Badaracco took part in those votes.

The major release was 56.0.0. Wang Xudong opened the RC1 vote, which drew 16 messages, and then an RC2 vote. Andy Grove gave RC2 a binding +1 and confirmed it fixes an FFI issue that was blocking datafusion-python from upgrading. That detail matters. DataFusion's Python bindings are how many data teams meet the engine, and an FFI blocker cuts them off from new releases.

The most important vote, though, was about growth. Gabriel Musat called a vote to accept datafusion-distributed as a DataFusion subproject, hosted in a new apache/datafusion-distributed repository. DataFusion Distributed is a Rust library that adds distributed execution to existing DataFusion apps. It stays close to native DataFusion and leaves worker management, networking, and deployment to the app that embeds it.

Gabriel was clear about how it fits next to Ballista. Ballista is a full distributed engine with its own scheduler, executors, and operations tooling. datafusion-distributed is an embeddable library for apps that already manage their own workers. Gabriel and Jayant Shrivastava plan to keep maintaining it inside the community. Acceptance is subject to ASF IP clearance before the repository moves. Support was strong. Andy Grove and Jeffrey Vo cast binding +1 votes. Andrew Lamb, Wang Xudong, Adrian Garcia Badaracco, and L. C. Hsieh also voted +1. Alessandro Solimando, who has reviewed PRs on query planning and adaptive query execution for almost a year, said the project clears the quality bar for design, implementation, and maintenance. Xuanwo and Martin Grigorov added non-binding support.

This pattern keeps repeating across the Rust data ecosystem. DataFusion is a toolkit, not a product, and the community keeps choosing composable libraries over a single monolithic engine. A distributed library that slots into existing apps fits that philosophy well.

Parser plans and a call for benchmarks

Andy Grove posted a draft PR that moves DataFusion to a proposed sqlparser-rs 2.0, PR 27304, with all tests passing except a dependency check that waits on a published crate. He opened an issue to discuss the parser and dialect design. Adrian raised the maintenance question directly and said he is not ready to take on sqlparser-rs maintenance himself. Martin Grigorov also joined. A parser rewrite is a long-term commitment, and the thread shows the community thinking about who carries it.

Andrew Lamb sent a call for real-world benchmarks. He wants to focus optimization work on cases that matter in practice and asked users to describe any workload they want faster in detail on issue 27330. If you run DataFusion in production, this is a direct line to the people deciding what gets faster next.

Apache Ossie (Incubating)

Relationships, joins, and what consumers guess

Ossie's list ran 81 messages, and the theme was precision. The semantic model spec is now meeting real engines and BI tools, and every gap shows up as two consumers guessing differently.

The busiest thread was a GitHub discussion on how semantic engines should interpret Ossie relationship definitions, with 10 messages. Julian Hyde added a caveat to an earlier point: his answer assumed the database has referential integrity. Christian Eubank replied that referential integrity helps if the model is anchored at Orders, but BI users often expect a query grouped by state name to include states with no orders at all. Referential integrity resolves the rows that exist. It does not say what to do with the dimension rows that have no facts.

Damian Waldron raised a related gap in a thread titled "Join type is not carried by a relationship, and consumers are guessing differently". He asked whether that is settled, since consumers fill the gap in different ways, and at least one of them was losing rows. He positioned it next to Matthew Mullins's explicit relationship cardinality proposal, which covers multiplicity but not join type. Matthew agreed to drop OneToMany and keep the existing from/to semantics after JB's points on backward compatibility and the risk of silent fan-out. He also accepted PascalCase for enum values to match other enumerations. Damian also posted a thread on model expressions versus query semantics, noting that for ThoughtSpot some expressions are not model-only at all, because the query completes their meaning. He said he measured this, and it changed what their converter emits.

Core spec details

Several core spec threads tightened definitions. Julian Hyde, Harel Shein, Chris Eubank, Jakub Moravec, and Matthew Mullins discussed what Dataset.source should guarantee. Harel, from the OpenLineage technical steering committee, introduced himself and noted that OpenLineage settled on almost the same structure: dataset identity as a namespace plus a name. Julian listed the lineage questions a business user asks, like when a dataset was last updated, how often it refreshes, and which Parquet files fed a query result. The overlap with OpenLineage is a strong signal. Semantic layers and lineage systems are converging on the same identity model.

Aleksandr Kosachev opened a thread on time zones. The spec defines CURRENT_DATE without a zone, so CAST(created_at AS DATE) on a zoned timestamp returns different dates depending on each engine's session or server zone. He proposed landing the Layer 1 primitives first, starting with AT TIME ZONE, and then a model-level default zone. JB agreed with the two-step plan and argued for a fixed rule on skipped ...