Streaming to Apache Iceberg: Understanding the Latency Physics
A 2026 technical guide to the physics of streaming into Iceberg, covering commit costs, small-file risk, and the latency trade-offs between Flink and Spark.
By 2026 the real engineering question is no longer whether to adopt Iceberg, but how fresh streaming pipelines can keep Iceberg tables and what to do when that freshness isn't enough. That shift exposes three physical realities baked into Iceberg's design: data only becomes visible once a commit publishes a new snapshot; every commit carries real cost because it writes fresh metadata and performs an atomic catalog swap; and frequent commits accumulate small files that degrade query performance over time. These constraints mean every streaming-to-Iceberg product on the market is really answering the same three questions differently: who buffers the data, who decides commit timing, and who handles maintenance.
Among open-source options, Apache Flink remains the reference for low-latency ingestion, aligning its checkpoint cycle with Iceberg snapshot commits to deliver seconds-level freshness with exactly-once guarantees. Its Flink CDC capability, which streams database change logs into Iceberg via equality deletes, makes it the standard tool for near-real-time database mirroring—though this convenience comes with an equality-delete backlog that slows reads until maintenance catches up, a burden that v3's deletion vectors are starting to ease.
Spark Structured Streaming offers a pragmatic middle ground through micro-batching, delivering minute-level freshness while minimizing operational friction for teams already running Spark. Its trigger interval gives a single clear dial between freshness and file health, but it can't match Flink's most aggressive latency, and always-on clusters often sit underutilized between triggers.
Across both approaches, the critical takeaway is that ingestion must be paired with a dedicated maintenance pipeline for compaction and snapshot expiration—otherwise the freshness gained becomes unqueryable. Understanding this physical framework lets engineers evaluate any new streaming-to-Iceberg product that emerges, since the underlying constraints never change.