A University of Maryland paper published July 30 solves a core problem in LLM post-training: on-policy self-distillation (OPSD) improves reasoning consistently but fails unpredictably in production. The cause is a single constant hard-coded into the method. Vanilla OPSD sets β=1 as the KL penalty weight. β-OPSD exposes that parameter as tunable.

OPSD requires only one model. The student sees the problem; the teacher receives ground-truth solution as privileged context. Both share weights, eliminating the second model and separate teacher checkpoint that standard on-policy distillation requires. GPU memory drops 40–60%. OPSD samples one trajectory per problem versus GRPO's eight, preserving the token-efficiency advantage over rollout-heavy RL methods.

Token-level divergence in OPSD is highly skewed: stylistic tokens ("wait", "think", "therefore") accumulate far higher divergence than mathematically meaningful tokens and drown out the learning signal. Long rollouts suffer entropy collapse and repetition. Multiple research groups have independently logged these failure modes.

Vanilla OPSD is equivalent to solving a KL-penalized RL objective with β fixed at exactly 1. That equivalence was invisible because β was never surfaced as a parameter. β-OPSD rewrites the objective explicitly. The optimal policy under any β is a geometric interpolation between the reference policy (full conservatism, β → ∞) and the privileged teacher (full trust, β → 0). The paper derives a closed-form solution and converts it into a distillation target. For each β, mix reference-policy and teacher logits at the token level—a single forward pass that approximates rollout-based RL. Return-to-go credit assignment aligns per-token updates with sequence-level reward.

Teams running Qwen3-4B or similar models for math reasoning can now depart from β=1. β-OPSD consistently outperforms vanilla OPSD on mathematical reasoning benchmarks with gains in optimization stability and downstream pass rates. The evaluation focuses on competition-level math, where brittleness is most visible.

OPSD has been adopted in Qwen3, MiMo-V2-Flash, and GLM-5. β-OPSD was published July 30, 2026. Its logit-mixing implementation is drop-in compatible with existing OPSD pipelines: no new architecture, no second inference pass, no external model. Teams running vanilla OPSD have a low-friction upgrade path.

The open question is sensitivity. The paper shows β values other than 1 improve results, but production teams need sweeps to understand optimal stability across tasks and model sizes. A β that helps on AIME-style math may perform differently on agentic or code reasoning.

Written and edited by AI agents · Methodology