OPD Outperforms GRPO for Long-Horizon Agent Planning

A team from the Chinese Academy of Sciences published the first systematic controlled study on how multi-turn long-horizon planning emerges during foundation model agent training. The paper, "The Physics of Multi-Turn Long-Horizon Planning," isolates each training stage in a synthetic environment, eliminating confounds from Internet-scraped data and providing teams a reproducible recipe for training smaller agents that plan across sequential steps.

The authors structure training into three phases. In pre-training, explicit world model construction via chain-of-thought state transition modeling produces stronger long-horizon generalization than teaching atomic skills alone. A small quantity of long-horizon trajectory data improves student performance, but suboptimal trajectories cause disproportionate damage—errors amplify over long horizons.

In post-training, the paper compares GRPO (group-relative policy optimization) against on-policy distillation (OPD). Using mutual information, the authors separate general planning patterns from task-specific knowledge and map three post-training regions: unnecessary (the model handles it), effective (post-training improves it), and unsupported (post-training cannot bridge the gap). OPD shows broader effectiveness than GRPO under low-quality and long-horizon conditions because it provides consistent gradient updates. GRPO's sparse scalar reward degrades under the same settings.

The third stage, multi-teacher on-policy distillation (MOPD), matters most for teams composing agents from multiple specialist models. MOPD succeeds when teacher planning patterns are compatible; cross-environment generalization follows. Partially shared patterns yield continual learning without catastrophic forgetting. Conflicting patterns cause severe interference with no middle ground—the authors frame this as a hard constraint on viable teacher combinations.

The controlled environment is the methodology's core strength. Prior work on multi-turn OPD—TCOD, Guided-OPD, and ReOPD—diagnoses failures on standard benchmarks like ALFWorld, ScienceWorld, and WebShop, where Internet pretraining data confounds variables. Men et al. isolate data format, distribution, and quality independently, making their findings about atomic skills versus long-horizon data and suboptimal-trajectory contamination causally attributable rather than correlational.

The paper does not benchmark against production models or report leaderboard accuracy. It trades external validity for internal control. Teams seeking drop-in comparisons will not find one. The paper provides a mechanistic account of training interactions: pre-training data quality gates post-training headroom; OPD applicability depends on pre-training trajectory quality; MOPD compatibility must be evaluated before multi-teacher setup.

Architecture teams should filter suboptimal trajectories aggressively at pre-training, use OPD over GRPO when planning horizons are long or trajectory quality is mixed, and audit teacher planning-pattern overlap before multi-teacher distillation. Conflicting patterns will harm performance, not average out.