Two Workers Both Held the 'Only One' Lock. The Clock Was Never Synced
A billing-reconciliation job ran twice due to clock drift. Solutions include auto-renewing locks and monitoring NTP synchronization.
A nightly billing-reconciliation job mistakenly ran twice on two workers due to unsynchronized clocks. The Redis lock logic was sound, but a 9-second drift in local clocks led to both workers believing they held the lock simultaneously. To mitigate such issues, it is recommended to auto-renew locks, implement fencing tokens for writes, and monitor NTP synchronization status to prevent future discrepancies.