How Airbnb Broke Circular Dependencies in Its Observability Stack
Airbnb engineers explain how they eliminated circular dependencies in their metrics pipeline using dedicated Kubernetes clusters and a custom Envoy-based networking layer.
Airbnb's engineering team details a subtle but critical flaw in their observability stack: their metrics pipeline depended on the same Kubernetes and Istio service mesh infrastructure it was meant to monitor. This circular dependency meant dashboards and alerts could go dark exactly during outages—when visibility mattered most.
The fix came in two layers. For compute, rather than choosing between fully shared production clusters or self-managed Kubernetes, the team adopted dedicated clusters still administered by the Cloud team but isolated from product and infrastructure workloads. For networking, instead of sharing the same Istio data plane as business traffic, they built a custom Envoy-based Layer 7 ingress layer that separates telemetry traffic, uses tenant-header routing across Airbnb's 1,000+ services, and shields the ingest path from service-mesh failures.
The approach offers a valuable lesson for engineers building systems at scale: observability infrastructure must be more reliable than what it observes. Over-reliance on shared platforms like Kubernetes or service meshes can strip away visibility precisely when outages strike. Airbnb's solution kept operational overhead manageable by clearly scoping responsibility—guaranteeing redundancy only for internal customers rather than across all external fault domains.