How Grab Scaled Its Data Lake by Adopting Apache Iceberg

Grab has detailed its migration from a Hive Parquet-based data lake to Apache Iceberg after directory-based storage and the Hive Metastore became scaling bottlenecks. Issues included metadata listing latency growing with partition count, severe small-file fragmentation in some datasets, heavy manual overhead for partition management, and a persistent drift between the catalog and actual storage state.

The migration delivered measurable gains: roughly 10x faster query runtimes on a high-traffic dataset via Z-ordering (down from 70 to 6 seconds), up to 95% lower daily S3 API costs on a heavily queried table, and roughly half the compute resource usage on another workload.

To address the developer-experience fallout of running mixed table formats, Grab built and open-sourced UnifiedSparkCatalog, a Spark catalog abstraction that transparently routes queries to the correct format-specific catalog (Iceberg, Delta, Hudi, or Hive) without requiring engineers to know or reference the underlying format.

For engineers operating large-scale lakehouses, the writeup offers a concrete look at the operational tradeoffs, performance wins, and design decisions involved in migrating a production data lake to a modern table format.