The Serverless CPU-Throttling Bug That Freezes Background Tasks

A webhook receiver on Cloud Run used FastAPI's BackgroundTasks to return a fast acknowledgment while launching the real job afterward. Sometimes the job showed up minutes later, sometimes never — until an unrelated request hit the service, at which point the 'stuck' job would suddenly complete. That behavior pointed to a frozen process rather than ordinary slowness.

The root cause was Cloud Run's default request-based CPU allocation: the instant a response is sent, CPU is throttled to near zero, and any code scheduled to run after that point stays suspended until the next request arrives. The event loop doesn't crash — it simply isn't scheduled — and since nothing throws or logs an error, the bug is nearly invisible. It also inverts the usual pattern, appearing on quiet traffic and hiding on busy days, which makes it easy to mistake for network flakiness.

The fix was to await the latency-critical call before responding instead of backgrounding it, rolling back the dedup marker on failure and surfacing a real 5xx so the sender's retry logic kicks in. When the pre-response work is inherently too slow to fit an ack deadline, the durable fix is switching the service to instance-based CPU billing and persisting that setting in infrastructure-as-code rather than as a manual CLI patch.

The case is a reminder that 'run it after the response' carries no CPU guarantee on request-billed serverless platforms. The practical testing lesson is that only the failure path — checking for a 5xx and a cleared dedup marker — actually distinguishes inline execution from backgrounded execution, since most test clients run background tasks synchronously before assertions run.