Chasing True Zero-Downtime Redis Sentinel Failover on Kubernetes

A Reddit post details one team's effort to eliminate downtime during Redis Sentinel failovers on AWS EKS, running a Bitnami Helm chart setup with a Node.js backend using ioredis. The team discovered that the default preStop hook's CLIENT PAUSE command froze their application for roughly 20 seconds during pod termination, triggering widespread TimeoutErrors. They replaced it with a custom script that immediately triggers SENTINEL FAILOVER and cleanly closes TCP connections, while configuring ioredis with maxRetriesPerRequest: null and enableOfflineQueue: true so that commands are buffered in memory during the outage and flushed once a new master is discovered. In testing, failovers complete in 2-5 seconds with no visible errors to end users, just a slight latency bump.

The open question is whether this holds up under real production load. The author worries that under heavy traffic, large datasets, and DNS propagation delays, Sentinel elections could stretch failover time to 10-15 seconds or more — at which point the in-memory command buffer could balloon, risking OOM crashes or severe latency spikes when thousands of queued commands are flushed at once. They're asking the community how others handle these Redis HA edge cases at scale, whether accepting occasional latency spikes is standard practice, and whether migrating to a managed service like AWS ElastiCache is the only reliable way to sidestep the problem entirely.

The discussion offers a practical lesson for anyone self-hosting Redis Sentinel on Kubernetes: preStop hook behavior, client-side buffering strategies, and unpredictable failover durations all carry real production stability risks. It also highlights the recurring trade-off between operating your own HA infrastructure and offloading that complexity to a managed service as scale increases.