Retry is not a loop, it's a data structure
Why treating retries as simple loops instead of data structures silently corrupts ordering, causes duplicates, and breaks distributed system histories.
Most engineers write retries as a simple loop: try, wait, retry with backoff, give up after N attempts. That's correct for a single isolated operation. But once you have a stream of operations where only some fail, the ones that succeed on retry can land out of order relative to items that never failed at all. The result: a log reading 1, 2, 4, 5, 6, 7, 8, 9, 3. Harmless for cache warmups, but catastrophic for timelines, ledgers, chat histories, or audit trails — systems that quietly rearrange history under failure, in ways no test will catch.
The piece proposes three questions to ask before writing any retry: Does order matter? Does duplication matter? Does staleness matter? A 'yes' to any of these means the retry needs machinery — sequence numbers, idempotency/dedup keys, or TTL-based staleness checks — turning what looked like a loop into an actual data structure managing state and invariants.
This distinction marks one of the quiet lines between mid-level and senior systems engineering. Mid-level code asks whether the call succeeded; senior code asks whether the story the system tells will still hold true after a bad network day. As more tooling emerges to reconstruct event histories — such as systems that trace which lines of code were written by humans versus AI agents — the correctness of retry ordering becomes foundational, not incidental.