Netkit: Closing the Linux Container Networking Performance Gap

A new paper from the Cilium team examines why container-to-container traffic underperforms compared to bare host networking. Benchmarks show that two processes in the same network namespace achieve 31% higher throughput than two containers on one host, and non-containerized hosts see a 26% edge over the wire — overhead traced to veth devices forcing every packet through a per-CPU backlog queue.

To fix this, the authors combine bpf_redirect_peer, a BPF helper merged into Linux 5.10, with netkit, a new device type upstreamed in Linux 6.7. Together they let packets cross network namespace boundaries without hitting the backlog queue twice, while netkit's primary/peer device split keeps BPF programs safe from being detached inside the container and gives them access to the host's routing table.

Testing on Cilium v1.19.5 with TCP_RR and TCP_CRR shows netkit-enabled pods matching host-networking throughput, and even beating it on CPU usage for TCP_CRR because connection-tracking overhead is skipped entirely. The optimizations are already available in Cilium via bpf.hostLegacyRouting=false and bpf.datapathMode=netkit.