Apache Lakehouse Weekly: Governance Debates Across Iceberg, Parquet
Apache Iceberg, Parquet, and Polaris communities debated format governance this week: statistics ownership, file-level access control, and primary key support.
This week the Apache lakehouse community focused on how open data formats should evolve without breaking existing tables. Parquet opened a vote to adopt versioned releases for breaking changes, while Polaris withdrew its semantic model API vote to align with the newly incubating Ossie project. Meanwhile, Iceberg Rust 0.10.0 moved from RC2 to RC3 after verification issues surfaced, underscoring the caution warranted given the library's growing role underneath PyIceberg-core, DataFusion, and JVM-free services.
The most consequential Iceberg debate concerned table statistics: when multiple engines like Spark, Trino, and Impala write stats for the same snapshot, one can silently overwrite another's work, and no consensus emerged on a fix. A separate proposal to extend REST catalog access delegation to the file level, using pre-signed URLs for partition-scoped sharing, would close a real gap with proprietary platforms' fine-grained access controls. The long-standing absence of primary keys in Iceberg also resurfaced, with production experiments describing key-aware write semantics for CDC and mutable-data workloads.
Taken together, these threads reflect the unglamorous but critical work of governing format evolution across a multi-engine ecosystem, work that directly shapes performance, security, and CDC support for anyone building on open lakehouse formats.