Engineering webhooks that never get lost: queue, retry, idempotency
A practical breakdown of reliable webhook delivery architecture: fast ack, queuing, idempotency, exponential backoff, dead letter queues and replay for payment integrations.
Anyone who has integrated a payment gateway (Stripe, Hotmart, Kiwify) eventually hits the same wall: a webhook that silently gets lost, or one that fires the same event twice. The source breaks down why reliable webhook delivery is a classic distributed-systems problem. A naive synchronous handler (receive, validate, process, respond) risks exceeding the sender's timeout window, causing the gateway to mark the event as failed even though your system may have completed it — an ambiguity that leads to real data loss.
The fix is layered: separate ingestion from processing by persisting the raw payload and returning 2xx immediately, then handling the real work asynchronously via a worker reading from a queue. That queue, however, introduces duplicate-delivery risk, since gateways resend events they believe failed. The answer is idempotency keyed on a stable resource identifier (not a delivery-attempt id), enforced with a database UNIQUE constraint so repeated inserts are simply ignored.
On the outbound side, delivering to the client's endpoint faces the mirror problem — the destination itself may be down. Instead of hammering it with immediate retries, exponential backoff (growing intervals like 1min, 5min, 15min, 1hr) gives the destination room to recover, with events falling into a Dead Letter Queue after repeated failures for manual inspection. Logging both inbound and outbound payloads enables diagnosis and bulk replay after outages, and the whole pipeline must stay lightweight enough to absorb traffic spikes like Black Friday without collapsing. The author mentions building HookSafe, an early-access tool implementing these patterns as a drop-in reliability layer between gateways and client servers.