Meta cuts ads serving p99 latency 28% with custom kernel scheduler built on sched_ext; 3.3MW power savings

Meta's engineering team deployed a custom Linux kernel scheduler using sched_ext (an open-source, BPF-based extensible scheduling framework in kernel v6.12) to optimize its ads serving fleet, which handles 5 million requests/second and 400 billion ads/day. The change from generic Linux schedulers (CFS, EEVDF) to a workload-specific policy reduced p99 latency on the ads retrieval stage by 28%, cut power consumption by 3.28 megawatts across the fleet, and improved ads ranked by 1.1%.

The optimization addresses a specific operational blocker: a kernel upgrade from v6.4 to v6.9 introduced latency regressions via the new EEVDF scheduler. Rather than staying on older kernels, Meta built a custom BPF-based scheduler that partitions CPUs into latency-critical request paths and background work, dynamically balancing load while improving L3 cache locality. Two follow-on updates (shipped as user-space BPF program changes, not kernel rebuilds) delivered an additional 60% p99 latency reduction and 18% timeout-error reduction.

For architects managing large-scale serving infrastructure: sched_ext represents a shift from fixed kernel schedulers to programmable, domain-specific scheduling policies. Meta's results (28% → 88% total p99 reduction) show the business impact of scheduler-level optimization on latency-sensitive workloads. The key insight: improvements ship in days rather than months because scheduler policy lives in user-space BPF, decoupled from kernel release cycles. This pattern matters for anyone building custom serving stacks or managing multiple kernel versions in production.