ALiBi Models Lose Positional Awareness in Long-Context Inference

A paper published August 4 identifies a failure mode hiding in every production deployment of ALiBi-based models: the encoding's linear bias scaling underflows floating-point precision at long contexts, zeroing out attention weights across many heads. The research on 148M-parameter models confirms the defect is present in state-of-the-art pretrained models shipped today.

ALiBi (Attention with Linear Biases) eliminates traditional positional embeddings by adding a head-specific slope penalty directly to attention scores before softmax. Each head receives a unique slope: steep slopes for short-range heads, shallow slopes for long-range heads. The penalty grows linearly with token distance. In FP32 this works as designed. In the half-precision formats that virtually every production inference stack uses, it breaks silently.

As sequence length grows, the bias term for steep-slope heads becomes so large a negative number that it underflows past what FP16 or BF16 can represent. After softmax, those heads assign near-uniform attention to distant positions. They are not dead weights—parameters are intact, the forward pass completes without error, loss metrics show nothing abnormal—but they have lost positional discrimination. SambaNova's engineering team documented this in 2023: at sequence length 8,192 in FP16, the last 20 positions of BLOOM-7B attention head 0 collapse to 5 distinct values (every 3–4 positions get the same encoding). In BF16, with 3 fewer significand bits than FP16, those 20 positions all receive identical embeddings.

The Schröder et al. paper separates two effects practitioners conflate: out-of-context degradation (performance drop from unseen lengths) and underflow pathology (performance drop from arithmetic failure). Pretraining experiments show the failure impairs token retrieval tasks while causing minor effects on standard benchmarks. That gap is the danger: perplexity and MMLU evals give no warning, but passkey retrieval and needle-in-a-haystack tasks—the workloads driving long-context model adoption—degrade substantially.

A March 2026 paper from Palmer Schallon (arxiv 2603.09616) quantifies prevalence: 31–44% of attention heads across BLOOM 560M through 7.1B attend almost entirely to the beginning-of-sequence token, concentrated in the steepest ALiBi slope heads. Schallon shows targeted Q/K/V reinitialization with zeroed output projections recovers 98.7% of operational head capacity (242 to 379 of 384 heads in BLOOM-1b7) in two GPU passes. The recovered model outperforms stock BLOOM-1b7 by 25% on training perplexity (12.70 vs. 16.99). These heads are dormant, not redundant. Pruning them destroys recoverable capacity.

The Schröder team tested four training-time mitigations. Log-scaled distances—replacing the linear distance term with its logarithm—yield the most consistent improvements in passkey retrieval. Default ALiBi slopes remain a strong baseline for needle-in-a-haystack despite the failure, explaining why this has gone undetected: aggregate retrieval benchmarks can look acceptable while the mechanism is broken.

Architects running BLOOM, MPT, or any ALiBi-based model at contexts beyond a few thousand tokens in fp16 or bf16 should know: attention heads are silently misfiring, standard evals will not catch it, and the fix requires retraining with log-scaled distances. There is no inference-time patch.