Kafka 3.7 vs 4.3: How linger.ms Defaults Shape Real Performance
A Kafka benchmark analysis showing much of the 3.7.2 vs 4.3.0 performance gap traces back to the linger.ms default changing from 0 to 5 ms, not core engine improvements.
A performance engineer benchmarking Apache Kafka 3.7.2 against 4.3.0 with Dimster found that under low load 4.3 showed higher end-to-end latency, while under heavy load it produced a far smoother p99.9 latency profile. The root cause traces back to Kafka 4.0 changing the producer's default linger.ms from 0 to 5 ms — a tweak that boosts batch efficiency under sufficient load but merely adds idle waiting time when traffic is light.
The analysis shows that the real driver of batching behavior is the per-producer, per-partition send rate rather than aggregate cluster throughput. In a 5K records/s test, each producer sent roughly one record every 24 ms, making a 5 ms linger essentially useless; in a 100K records/s test, the inter-arrival gap shrank to about 1.2 ms, letting the same linger value meaningfully grow batch sizes and cut p99.9 latency from around 700 ms down to roughly 8 ms.
A follow-up experiment with keyed workloads confirmed that a higher linger.ms of 20 delivers major gains under heavy, partition-spread traffic, and that at that setting 3.7.2 and 4.3.0 perform almost identically — indicating the apparent version-to-version improvement is mostly explained by one client default, not core engine changes. The practical takeaway for engineers: watch actual producer batch sizes and per-partition send rates instead of assuming total throughput tells the full story.