Meta's Ad Model Hits 20% GPU Efficiency at LLM Scale

Meta's Generative Ads Recommendation Model (GEM) now runs at LLM scale on several thousand GPUs and has achieved 20–25% end-to-end Model FLOPs Utilization (MFU), double the prior baseline. The company also scaled total training FLOPs 4x over 12 months. The results were published August 3 in an engineering post by 15 members of Meta's ads ML infrastructure team.

LLM training infrastructure doesn't transfer to recommendation systems. GEM is a hybrid—trillions of sparse embedding parameters plus billions of dense transformer parameters—and its input data differs fundamentally from language. User activity histories vary wildly in length. Padding to the maximum sequence length wastes up to 50% of compute. Attention shapes are asymmetric in three ways: self-attention over long sequences with short windows, cross-attention with long queries but short key/value, and pooled multi-head attention with the inverse ratio. Standard Flash Attention kernels and FSDP recipes don't handle these patterns efficiently.

Meta built a custom kernel library. Jagged Flash Attention (JFA) eliminates padding waste by operating natively on variable-length sequences. Generalized Dot-Product Attention (GDPA) handles asymmetric q/k/v shapes. BlockAttention improves hardware saturation for structured access patterns. The team adopted MXFP8 for attention and MLP layers, but CTR and CVR prediction are numerically sensitive—naïve low-precision training causes quality regression, requiring custom stability work.

Scaling across thousands of GPUs created a second problem set. Meta's formula: E2E MFU = Local MFU × Scaling Ratio. Near-linear scaling requires compute to dominate communication, communication to be fully hidden behind compute, minimal activation recomputation, and balanced load across ranks. GEM violates all four. Trillions of sparse parameters generate heavy all-to-all communication. Long sequences push activations to memory limits, forcing recomputation. Jagged inputs create load skew that varies across ranks per step.

The solution: a topology-aware 5D parallelism scheme with SM-free collectives. Dense parameters use 2D FSDP combined with Expert Parallelism; sparse parameters use Fully Sharded 2D Model Parallelism. The critical detail is SM-free collectives—by routing collective operations through dedicated network hardware instead of Streaming Multiprocessors, compute and communication no longer compete for resources. This unlocks communication hiding at thousands-of-GPU scale.

The result is 20–25% E2E MFU—doubled from the baseline—while simultaneously scaling FLOPs 4x. LLM training typically achieves 35–50% MFU; a recommendation model crossing 20% at this scale with hybrid sparse/dense architecture represents a different engineering category.

For architects scaling hybrid sparse-dense foundation models, the standard LLM approach—vanilla FSDP, standard Flash Attention, naïve MXFP8—leaves efficiency on the table. The two design decisions worth adopting: separate parallelism strategies for sparse and dense parameters, and SM-free collectives to eliminate compute/communication contention.