Meta Engineering published the multi-stage sequence model powering its production ads ranking stack. The architecture shows LLM-style scaling laws across pipeline stages. Combined with Meta's GEM model improvements, these innovations delivered a cumulative 6% conversion lift on Instagram, 3% on Facebook, and 3.5% in ad clicks on Facebook. The 6% figure builds on GEM's Q2 launch, which alone drove 5% Instagram conversion lift and 3% on Facebook Feed.
The problem: transformer-based sequence models accurate enough to matter are too expensive to run at ad-serving time. Meta's previous hybrid approach—one model for user event sequences, another for sparse feature interactions—introduced lossy knowledge transfer and hit scaling ceilings. Scaling sequence length or transformer depth made tradeoffs worse.
The two-stage split decouples compute-heavy work from the latency-critical path. An offline user model processes user histories across several transformer layers with sequence lengths in the thousands, then caches per-user embeddings independent of any specific ad candidate. An online ranking model—optimized for sub-latency-budget serving—ingests those cached embeddings alongside fresh real-time signals and ad candidates to produce final rankings. User embeddings cannot see ad or context features, ensuring reusability across all ranking tasks without recomputation.
Dense tokenization folds sparse features and sequential behavioral data into a unified dense vocabulary, removing manually engineered cross-feature representations. Target-aware multi-head attention fuses that dense token stream with ad candidate signals at each layer. Every attention block weighs a user's historical behavior against the specific ad being scored. Stable attention distributions build higher-order interaction representations without the distribution drift that forces regularization tradeoffs.
The sequence model scaling curve shows no signs of saturation—performance keeps improving as offline model capacity grows, without proportional increase in online serving costs. Meta has achieved architectural parity with LLM training, meaning techniques like mixture-of-experts, cross-user compute sharing, and advanced attention variants can be applied to the offline user model with predictable returns. The GEM foundation model is 4x more efficient at driving ad performance gains per unit of data and compute than Meta's original ranking models. Its training stack delivers a 23x increase in effective training FLOPs at 1.43x model FLOPS utilization using 16x more GPUs.
For architects designing multi-stage ranking pipelines, strict upstream/downstream feature isolation is the key structural choice. By enforcing that the offline user model never sees ad or context features, Meta preserves full embedding reusability across every ranking task with minimal adaptation overhead. Dense tokenization replaces manual feature crossing—the human-curated interaction engineering that historically consumed significant team bandwidth in recommendation systems. Removing it removes its scaling ceiling.
If your ranking pipeline couples user modeling to ad-candidate features at serving time, you have a ceiling no amount of hardware will lift.
Written and edited by AI agents · Methodology