One hyperparameter rule captures most of µP's gains

University of Maryland researchers have identified why Maximal Update Parameterization (μP) outperforms standard parameterization (SP) in LLM training. The culprit: embedding layer learning rate. When training with AdamW, scaling the embedding LR with model width captures the majority of μP's transfer benefit and eliminates a critical bottleneck that degrades training stability at scale.

The paper "Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate" (Kalra & Barkeshli, arXiv 2605.21486) proposes three metrics for auditing hyperparameter transfer: (1) quality of the scaling law fit, measuring how cleanly optimal HPs follow a power law across model widths; (2) robustness to extrapolation errors, tracking loss degradation when small-to-large transfer is slightly off; and (3) asymptotic loss penalty due to parameterization choice.

The mechanism is simple. In SP, the embedding layer learning rate stays fixed as the model widens. This creates a bottleneck: the embedding sees a learning rate that becomes relatively too small at larger scales. μP multiplies the embedding LR by the width factor, smoothing training and unlocking better hyperparameter transfer. Other μP rules contribute, but this single change accounts for most of the gain.

The practical implication is direct. Standard practice runs a 200-sample random hyperparameter search on a 40M-parameter proxy model and transfers the winner to 7B or 70B. Under SP, the proxy's embedding layer sees a learning rate that becomes biased at scale. The wider the target model, the more misleading the SP proxy. Under μP, the proxy's hyperparameter recommendations remain valid across scales.

The paper also examines weight decay. It improves scaling law fit quality—curves become cleaner and extrapolation more reliable. But in the fixed token-per-parameter budget regime (standard in compute-constrained settings), weight decay reduces robustness to extrapolation errors. This creates a tradeoff: tune for cleaner scaling curves and accept higher transfer variance, or tune for robustness and accept noisier fits.

One limitation: the paper lacks production-scale validation. No latency, throughput, or GPU-hour accounting. Experiments are systematic but omit model sizes, hardware, and token counts. Teams quantifying compute savings from improved proxy runs must measure this themselves.

An open question concerns vocabulary size. Concurrent work (arXiv 2506.15025) shows that as vocabulary grows large relative to model width—standard in modern LLMs—the optimal embedding LR to hidden LR ratio shifts from μP's prediction of Θ(d) toward Θ(√d). Both findings confirm the embedding LR deserves explicit treatment. The correct multiplier depends on vocab-to-width ratio, which this paper does not model.

For teams using SP proxy sweeps, the immediate fix is adding a per-layer learning rate multiplier for the embedding equal to the width-scaling factor. This single change captures most of the μP benefit with minimal implementation overhead.