Greedy decoding from large language models produces different outputs in BF16 versus FP16 on identical hardware, with 49–100% of prompts diverging across six models and three benchmarks, according to an arXiv preprint published in September 2026. A single token flip often cascades into trajectory-level divergence, making determinism-critical deployments unreliable when precision formats vary.
The root cause lies not in accumulated error across the model's body layers but in the margin between the top two logits at the language model head. The paper's empirical error-propagation analysis found that 22 layers of accumulated body error do not distinguish whether a step will flip to a different token; the outcome depends primarily on the top-two logit margin relative to the directional perturbation between the top-two candidates. This finding led to five testable predictions about intervention outcomes, including a counterintuitive one: applying more FP32 compute across a broader scope makes agreement worse, not better. Experiments matched all five predictions.
The mitigation the authors evaluated is selective FP32 LM head recomputation, triggered only when the margin between top-two logits falls below a threshold. On A10G hardware, this approach delivers +22–36 percentage points of exact agreement recovery at less than 4% latency overhead in low-batch (batch size ≤4) single-stream inference. Performance degrades on other hardware: the same method recovers +12–21 pp on L4 and A100. The applicability boundary is narrow: the method's benefit vanishes when body-originated error dominates, including at batch size ≥8 and under end-to-end FP8 quantization in the paper's tests.
The intervention is a partial mitigation rather than a universal determinism guarantee. The authors map the applicability boundary across six models (1.1B–7B parameters, four families, with divergence additionally characterized at 12B) and four batch sizes, and hypothesize that training-time precision stability is a determining factor in whether the method will work. At higher batch sizes or under aggressive quantization, the selective recomputation strategy fails because the error originates in the model body rather than at the head, making the head-level fix ineffective.
For architects deploying models where output determinism matters—compliance systems, reproducible research pipelines, or safety-critical inference—the selective FP32 head recomputation offers a concrete tool with clear operational boundaries: it works at batch size ≤4 on A10G with minimal latency cost, but does not generalize to higher concurrency or end-to-end FP8 quantization. Teams should measure divergence on their own hardware and batch profile before committing to a single precision format, and treat greedy decoding as precision-dependent rather than deterministic.