Muon's ability to accelerate grokking in transformer training has a shadow side: a new empirical study published 7 August 2026 by IBM Research shows that models trained under the standard Muon/AdamW hybrid split grok modular arithmetic rapidly, then collapse. All nine optimizer configurations tested on (a+b) mod 113 grokked, then lost generalization. Across five seeds, the selected AdamW reference recovered but still dropped below accuracy threshold on four of them, hitting 27.59%.
The failure is not a corner case. Authors Ali Janati, Kaoutar El Maghraoui, Andrei Kanavalau, and Anass Belfatmi tested the same setup across two moduli, two model widths, two training-data fractions, modular subtraction, and multiple depths. Every variant collapsed. The root cause is structural: the representation layer and output readout are jointly constrained only up to an invertible linear map that cross-entropy loss never resolves. There is no unique solution the gradient is trying to find, so neither parameter group is anchored to anything fixed.
The divergence becomes acute once the model solves the training set. The gradient norm drops to order 10^−6, effectively near-zero. The two optimizer groups respond oppositely: Muon's step-size elasticity is −0.03 while AdamW's is +1.5. Muon moves its parameter group 8.0 times faster per parameter. Muon continues reshaping hidden weight matrices while AdamW-managed embeddings and output head drift at a fraction of that rate. The representation-readout interface, already ambiguous by construction, is now pulled apart by mismatched update velocities.
| Property | Muon (hidden weight matrices) | AdamW (embeddings + output head) |
|---|---|---|
| Step-size elasticity | −0.03 | +1.5 |
| Relative update speed (per parameter) | 8.0× faster | 1× (baseline) |
| Post-grokking behavior | Continues reshaping hidden weights | Drifts at fraction of Muon's rate |
| Gradient norm at collapse onset | ≈ 10⁻⁶ (near-zero) | ≈ 10⁻⁶ (near-zero) |
The paper's cleanest evidence comes from a freeze experiment. Starting from bit-identical checkpoints, the researchers froze either parameter group over 451,400 post-grokking steps across five paired seeds. Freezing the embedding and readout group—the AdamW side—eliminated collapse entirely: zero sub-threshold evaluations in frozen runs, compared to 137–321 in unfrozen arms. One knob turns the failure off.
Stripping Muon's orthogonalization and normalization does not fix the problem. Without those operations, the internal Fourier representation collapses from 326 effective conjugate pairs to 4. That variant shows no recurrent collapse but fails terminally instead—a different catastrophe. Muon's geometry-preserving properties are necessary for the rich representation to exist at all. They also make the post-grokking velocity mismatch so damaging.
Fourier circuit analysis across 43 checkpoints and five seeds clarifies the mechanics. In normal operation, the task-aligned Fourier family reaches 100% accuracy on its own. In the collapse regime, two failure modes emerge: circuit failure, where the task-aligned family stops solving the task outright, and masking, where it remains perfect but is outvoted by a near-equal adversarial remainder—the full model reaches 45.85%. Rescaling the task-aligned circuit restores 99.9%. One critical detail: through an abrupt collapse, the standard Fourier support and power-distribution cosine (0.9899) remain stable. The standard diagnostic does not flag failure before it occurs.
| Scenario | Full-Model Accuracy | Task-Aligned Circuit Status | Standard Fourier Diagnostic |
|---|---|---|---|
| Normal operation | 100% | Solves task alone at 100% | Stable (cosine = 0.9899) |
| Circuit failure | Below threshold | Stops solving the task outright | Stable — does not flag failure |
| Masking (adversarial remainder outvotes circuit) | 45.85% | Perfect (100% alone), but outvoted | Stable — does not flag failure |
| After rescaling task-aligned circuit | 99.9% | Restored | N/A |
This matters for anyone running Muon in production. The standard split—Muon on hidden weight matrices, AdamW on embeddings and output head—is deployed in Moonlight and reportedly used in Kimi's >1T-parameter training runs. An earlier paper (Wang, 2026) showed that reducing the Newton-Schulz iteration count from five to one makes the grokked solution fragile at higher learning rates. The new work shows instability is possible even with the canonical five-iteration setup, driven purely by post-grokking velocity asymmetry.
The surgical fix the paper demonstrates is to freeze the embedding/readout group once the model solves the training set. Whether that generalizes beyond modular arithmetic to large-scale pretraining remains an open question. What is no longer open: Fourier-spectrum monitoring will not catch this collapse, and the standard assumption that grokking implies a stable generalized solution is wrong.