The New Physics of Columnar Storage: Parquet, Lance, Vortex, Nimble

For a decade the file format layer was settled ground, with Apache Parquet dominant across analytics. That stability has ended: AI workloads now demand point lookups into billion-row vector datasets, GPU-speed feature extraction, and multimodal blobs alongside structured columns, breaking assumptions baked into Parquet's 2013 design. Meanwhile NVMe storage made decompression, not transfer, the new bottleneck.

A research wave, BtrBlocks, FastLanes, ALP, and FSST, demonstrated that chained lightweight encodings (dictionary, run-length, bit-packing, delta) can match heavyweight codecs like Zstandard on compression while decoding at memory speed in SIMD-friendly patterns. This economic inversion, decode cost overtaking transfer savings, is the physics driving new entrants: Lance and Vortex from AI infrastructure startups, and Nimble, open-sourced by Meta from its ML platform.

For engineers, the takeaway is concrete: file format choice is no longer just about ecosystem compatibility, but about matching physical encoding strategy to workload shape, whether that's batch scans, random access, or GPU-fed training pipelines. Understanding these five design axes, layout, encoding, metadata, access pattern, and specification contract, is becoming essential to modern data infrastructure decisions.