Random Access Parquet: Fast Point Queries Over the Data Lake

Companies like Spotify need low-latency access to per-user data for both online services and LLM-powered agents, but at exabyte scale it's uneconomical to keep everything resident in a KV store like Bigtable or DynamoDB. Lake storage itself has gotten fast — GCS delivers 30-100ms, and S3 Express One Zone or GCS Rapid Storage reach single-digit milliseconds — but distributed SQL engines like Trino and BigQuery add seconds of planning and scheduling overhead even for a single-row lookup, since they're built for analytical throughput, not interactive point queries.

Random Access Parquet (RAP) closes this gap with an external index that maps keys directly to file locations, replacing the chain of dependent in-file reads (footer parsing, row group metadata, key column scans, page indexes) with a single precomputed lookup followed by precise, parallel ranged reads. The index lookup is O(1), and RAP works on existing Parquet files already shared by ML pipelines, notebooks, and batch analytics — no special preparation required.

Index size scales predictably — gigabytes per terabyte indexed, terabytes per petabyte — and distributes naturally via hash bucketing. Write-time techniques like sorting by key, hash bucketing, co-grouping, and coarser partitioning can further shrink read volume and round-trips, at the cost of partition-pruning granularity for batch queries. For engineers, the payoff is avoiding duplicate serving-layer copies: the same Parquet files can support both batch analytics and low-latency point lookups.