Netflix's Real-Time Service Topology: Architecture and Lessons Learned

Netflix engineers detail how they built a real-time service dependency map across a massive distributed architecture. The first version worked flawlessly locally, but production exposed hard truths: Kafka consumers fell behind, instances ran out of memory, and garbage collection pauses ate up CPU — pushing the team toward a streaming-first design.

At the core is a three-stage reactive pipeline that propagates backpressure upstream whenever downstream storage can't keep up, allowing the system to degrade gracefully under load instead of dropping data or crashing. Network, IPC, and tracing data are also split into physically separate storage layers, each optimized for its own throughput and query patterns, then merged in parallel at query time with sub-second latency.

The three-stage distributed aggregation pipeline specifically tackles a subtle but critical problem: raw network flow logs only capture individual hops through load balancers and proxies, not true application-to-application dependencies. By resolving these intermediaries, the system produces a clean topology refreshed within tens of minutes — a major leap over traditional hourly or daily batch processing.

For engineers building distributed systems, this is a concrete case study in backpressure design, layered storage isolation, and how streaming aggregation holds up under real production load.