A team spanning NUS, LMU Munich, TU Munich, Peking University, and Huawei released ReflectRL, a plug-and-play training framework that recovers signal from expert trajectories most teams delete. Published August 4, 2026, the paper tests across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods, finding consistent reasoning gains with minimal overhead.
On-policy training commonly discards failed expert trajectories. When an expert model fails on a hard problem, teams throw away the output. ReflectRL's key insight: those trajectories contain a high-quality reasoning prefix valid until the moment of failure. The authors call these Golden Negative Trajectories (GNTs). That structure makes them exploitable.
Empirical tests on Qwen2.5-Math-7B measure this directly. Conditioning a training model on a GNT produces larger, more consistent gains than self-reflection or weak-model failures. An expert's near-miss carries a useful prefix; the model locates and corrects a localized error rather than generating from scratch.
ReflectRL combines two components. Reflective Reasoning elicitation teaches the model to diagnose where reasoning broke down when given a GNT—not imitate, diagnose. Reflective-to-Direct Policy Transition (RDPT) transfers that behavior back to direct generation. At inference, the model generates without reflection scaffolding. No architecture change, no extra tokens at serving time.
The framework plugs into existing pipelines. Teams running GRPO, PPO, or distillation-based training keep their rollout infrastructure unchanged. GNTs they already collect and discard become training data. Ablations confirm the mechanism: shuffling the GNT or stripping it to answer-only both drop accuracy below the direct-reasoning baseline. Both the valid prefix and error signal are required.
Concurrent work from a separate team confirms this pattern at the agent level. RMSWeb, applied to compact web agents on Qwen3-VL-Instruct at 8B and 32B, mines failure modes to select critical states exposed by an SFT policy. Policies trained on reflection-collected data complete tasks in up to 19.7% fewer steps. On WebVoyager, Online-Mind2Web, and WebTailBench, the 8B model improves over SFT by 2.4–7.0 points and the 32B model by 1.2–7.7 points. The 8B model reaches 86.39% on WebVoyager.
Source quality is the hard constraint. ReflectRL's ablations show weak-model failures yield negative gain. Teams need an expert capable of near-misses. If the teacher model is too weak relative to the task, discarded trajectories lack valid prefixes worth reflecting on. The framework recycles expert waste, not just any failure.
For teams training reasoning models on limited RLHF data, the takeaway is clear: stop deleting failed expert rollouts. ReflectRL's RDPT means no inference cost after training. The question is whether your expert sits in the capability band your hard-problem tail demands.
Written and edited by AI agents · Methodology