Cache invalidation: solving stale data and stampede problems
An engineering look at TTL, event-based invalidation, versioning and single-flight patterns, and how cache stampede can take down your origin in production.
Stale cache data arises because the cache and the origin are two separate copies, and any synchronization mechanism—TTL, event-based invalidation, versioning—leaves a window where they can diverge. The real production risk isn't just users seeing outdated data for a few seconds; it's cache stampede, where thousands of requests miss simultaneously when a hot key expires and hammer the origin with the same expensive query, potentially exhausting the DB connection pool and taking the service down within seconds.
The piece walks through four invalidation mechanisms—TTL, event-based invalidation, versioning, and single-flight (request coalescing)—and how they're typically combined: TTL bounds staleness, single-flight collapses concurrent misses into a single origin call, and event-based invalidation cuts TTL short on writes. Code examples using Go's singleflight package and a Redis SETNX distributed lock illustrate the pattern in practice.
Five common production failure modes are detailed: stampede on a hot key's expiry, indefinite staleness when an event is dropped without a TTL backstop, thundering herd after a global FLUSHDB, inconsistent reads in L1/L2 layered caches after write-through, and forgetting a short TTL for negative caching. Each comes with root causes and fixes—single-flight, the XFetch algorithm, versioned key prefixes for gradual invalidation, and pub/sub-driven L1 eviction.
The final section covers metrics worth tracking—cache hit ratio, origin request rate, staleness distribution, lock contention—and how to alert on them in Prometheus/Grafana. The general rule: every cache entry needs a TTL, every stampede-prone heavy query should be wrapped in single-flight, and event-based invalidation should always have a TTL safety net.