A 24-Year-Old Linux Kernel TCP Bug Found via Stuck rsync Transfers

Skroutz gives each developer a writable, daily-refreshed database snapshot via a pipeline built on LVM snapshots, rsync, and ZFS. The setup scaled cleanly from 15 to 150 developers over nearly a decade, but occasionally an rsync transfer would hang indefinitely with no obvious cause, and the issue resisted reliable reproduction.

Engineers first ruled out network hardware — NIC offloading bugs, switch-level drops — before attaching strace and gdb to the stuck processes, which showed blocking syscalls but no forward progress. The real clue came from ss(8) output: the connection was sitting in a 'persist' timer state, meaning the sender had received a zero-window advertisement even though the receiver's buffer was actually empty and able to accept data.

That mismatch between an empty receive queue and a supposed zero window pointed to a genuine kernel-level defect in TCP's flow-control and zero-window-probe handling — a bug that had lived in the Linux kernel for 24 years. The case is a solid reminder of how low-level tools like ss, strace, and tcpdump remain essential for diagnosing mysterious production hangs, and how deep TCP internals knowledge still matters for infrastructure engineers.