Super Weights Training Fails on OLMo Models, Demolishing Sparse Fine-Tuning Strategy

New research has debunked the prevailing engineering belief in sparse fine-tuning by showing that training only the most critical parameters during inference can reduce accuracy to random-guessing levels. Experiments on OLMo-1B and OLMo-7B models demonstrated that isolating and updating between 100 and 8,192 Super Weights—parameters whose removal drastically affects model output—resulted in accuracy dropping to chance levels. Even expanding the trainable set to local neighborhoods of up to 36,000 coordinates did not improve results, and a ten-seed ablation confirmed the systematic collapse.

Super Weights originate from a 2024 Apple paper that identified scalar outliers in down_proj feed-forward layers, typically in early layers of LLaMA, Mistral, and OLMo families, acting as persistent governors of the residual stream. Pruning just one of these can cause perplexity to spike three orders of magnitude and zero-shot accuracy to collapse to guessing. This suggested a compression strategy: updating only these few coordinates should yield the most parameter-efficient fine-tuning. Subramanian, Akinfaderin, and Sehwag tested this on the Allen Institute's OLMo checkpoints, freezing all weights except Super Weight indices and running standard gradient descent.

The models failed to learn, and the failure was coordinate-specific, not a general sparsity penalty. Training an equal number of randomly chosen positions in the same down_proj layers, using identical optimizers and schedules, improved accuracy over the frozen baseline. Vanilla LoRA, updating every position in attention weight matrices through a low-rank decomposition using just 0.16 percent of parameters, succeeded. Applying the same low-rank update to the full down_proj matrices also worked. However, constraining those low-rank updates to the exact Super Weight indices produced statistically indistinguishable results from the isolated training collapse, confirming that gradients do not support learning at those specific coordinates when targeted in isolation.

No production evidence supports any sparse training method that successfully targets inference-critical coordinates; architects should treat Super Weight-aware training as disproven until reproduced on production fine-tuning pipelines with real downstream tasks. For inference stacks, weight-importance maps derived from pruning sensitivity or quantization robustness are unreliable guides for selecting sparse training targets. The Apple paper showed that preserving Super Weight precision makes round-to-nearest quantization competitive with SmoothQuant, so these coordinates remain critical for model serving. Yet, the same coordinate set is toxic for adaptation, and any pipeline using inference-time saliency to mask the optimizer—selective fine-tuning, sparse adapters, or partially frozen LoRA masks—is built on a false premise.

The risk is architectural. The experiments cover OLMo, while the original Super Weight catalog spans LLaMA 7B to 30B and Mistral-7B. It remains unverified whether the trainability gap generalizes across initialization schemes and feed-forward geometries, or whether alternative structured sparse formats such as block sparsity or N:M patterns can escape the trap. What is clear is that effective fine-tuning requires structured decomposition across entire layers, not sniper shots at individually important weights.

Parameter importance at inference time is a trap for selecting sparse training targets; use structured decompositions over whole layers even when updating only a fraction of the capacity.