Netflix's real-time service topology pipeline processes five million eBPF flow records per second across thousands of microservices, ensuring sub-second query SLA on dependency graphs, replacing static maps that became obsolete before engineers could access them.
The system divides observability into three physically separate graph layers, each stored in its own database partition. The network layer runs a FlowExporter sidecar on every Titus instance, capturing kernel-level flows using eBPF TCP tracepoints with less than one percent CPU and memory overhead. An IPManAgent writes IP-to-workload-ID mappings directly into eBPF maps at container launch time; FlowExporter reads them in-kernel, eliminating the delay-based misattribution that affected earlier Sonar event-stream approaches. These flow logs are sent to multi-region Kafka and processed by Apache Pekko Streams through three stages—initial batching, IP-to-service identity resolution, and graph DB write—into the network partition. A second partition ingests application IPC metrics delivered as Server-Sent Events, providing endpoint and protocol context for instrumented services. A third holds distributed traces in columnar storage for behavioral accuracy on sampled paths. A gRPC API exposes each layer individually or as a parallel merged union, supporting multi-hop traversal, filtering by availability tier, and pagination.
In production, the first version collapsed due to Kafka consumers falling behind, instances exhausting memory, and hot-partitioning concentrating roughly one hundred times the expected traffic on individual nodes. Garbage collection pauses consumed more CPU than the topology business logic itself. Unbounded in-memory queues caused out-of-memory crashes; drop-based flow control produced incomplete dependency graphs; and batch pipelines delivered data hours old, rendering them useless in an environment where services deploy multiple times per day.
The backpressure problem was resolved with reactive streams in Pekko Streams. When the final graph DB write stage slows, it signals upstream through the stream stages to pause the Kafka consumer. Records wait in Kafka until downstream capacity returns, allowing the system to degrade gracefully without data loss. Freshness improved from hours to "typically within tens of minutes." Historical queries rely on time-window aggregation rather than point-in-time snapshots, enabling engineers to reconstruct past topology states without the storage overhead of retaining full copies every time a service deploys.
Three independent layers mean no single signal is authoritative: eBPF covers every connection but lacks application semantics, IPC is limited to instrumented paths, and traces are sampled. Queries that merge across layers run in parallel, yet the Netflix TechBlog posts do not quantify consistency guarantees between partitions. The team also notes that reactive streams are harder to reason about than synchronous blocking code, imposing a debugging and onboarding tax. Future work includes injecting deployment and configuration change events into the graph and moving toward automated root cause analysis, though neither is in production yet.
The transferable pattern is not "deploy eBPF" but rather this: no single telemetry stream covers the full dependency graph, and the only thing that makes a five-million-record-per-second topology survivable is backpressure over Kafka—without it, the graph database or the eBPF agent is irrelevant.
Written and edited by AI agents · Methodology