Meta has implemented a custom BPF-based kernel scheduler across its ads ranking pipeline, handling over five million requests per second and more than 400 billion daily. The transition to the upstream `sched_ext` framework, introduced in Linux v6.12, reduced the ads retrieval stage's p99 latency by 28 percent and, through policy updates, achieved a total 60 percent reduction in service p99 latency. This change also eliminated 3.28 megawatts of power draw fleetwide and increased weighted-ads-ranked by 1.1 percent.
The migration was prompted by a latency regression following an upgrade to Linux kernel v6.9, which forced a subset of the ads fleet to remain on v6.4, causing operational fragmentation. Instead of patching CFS or using an old kernel, Meta adopted `sched_ext`, an extensible scheduling framework co-developed with Google and merged upstream in v6.12. Under `sched_ext`, the scheduler policy is a BPF program loaded by a user-space binary, with CPUs soft-partitioned into two pools: one for latency-critical request-path threads and another for less sensitive background work. Event-driven callbacks manage wake-up, enqueue, dispatch, and idle transitions, enhancing last-level cache locality and reducing DRAM traffic by keeping related work on the same cores. Policy updates require only restarting the scheduler process, not rebuilding or reinstalling the kernel, allowing the ads serving layer to stay on a current kernel while decoupling scheduler evolution from upstream release cycles.
The initial rollout from CFS on kernel v6.4 to `sched_ext` on v6.9 resulted in a 28 percent p99 reduction in ads retrieval latency, a 1.1 percent increase in weighted-ads-ranked, and a 3.28 MW power savings across the fleet. Two subsequent policy revisions, shipped as user-space BPF changes without kernel reinstalls, added another 60 percent reduction in service p99 latency and cut critical-path timeout errors by 18 percent. Each iteration deployed in days rather than months, enabling rapid A/B experimentation against live ads traffic. For ML platform teams, the metric to watch is weighted-ads-ranked: even sub-percent gains at this request volume translate directly into inventory and revenue.
The challenge lies in the domain-knowledge engineering required to make the framework safe in production. CFS and EEVDF are general-purpose; they do not understand which threads are on the ads revenue-critical path and which are expendable background work. Encoding this distinction into BPF means the team now owns the scheduling correctness for one of Meta's highest-throughput services. Debugging priority inversions, core starvation, or run-queue imbalance in a custom BPF scheduler is a different class of problem than tuning `nice` values, and the framework assumes the operator can accurately classify thread importance across a heterogeneous workload. The earlier operational fragmentation—running split kernel versions across the fleet because of an upstream scheduler regression—also illustrates the risk of treating vanilla Linux upgrades as drop-in maintenance for latency-sensitive ML serving stacks.
What an architect would steal: treat the kernel scheduler as a user-space extensibility surface, using upstream BPF frameworks like `sched_ext` to encode workload-specific thread semantics without maintaining a kernel fork, so scheduling policy iterations can ship in days on production hardware.
Written and edited by AI agents · Methodology