Renmin University of China researchers published CalibForge, an autonomous pipeline that synthesizes terminal-agent training tasks at scale. The system uses solver behavior to identify learnable tasks, then revises them. Result: 5,431 calibrated tasks that improve models by 24.71 percentage points on Terminal-Bench 2.0, 27.68 on SWE-bench Pro, and 30.04 on Doc2Repo.

BenchmarkDomainImprovement (pp)Note
Terminal-Bench 2.0Software / Science / Security / ML+24.71In-distribution
SWE-bench ProSoftware engineering+27.68Out-of-distribution
Doc2RepoCode documentation → repo+30.04Out-of-distribution
FIG. 02 CalibForge training gains across three benchmarks (percentage-point improvement over base models) — CalibForge paper (Renmin University of China)

The core insight: executability is not learnability. A task that every solver passes or fails trains nothing. Existing pipelines validate only that tasks run. CalibForge adds a calibration layer to measure whether tasks discriminate between solver capabilities.

Two calibration modes drive this. Multi-solver calibration routes candidates through a heterogeneous pool. If solvers disagree—some pass, some fail—the task is learnable. Contrastive solver calibration is surgical: designate a strong solver that must pass and a weak one that must fail. Tasks missing this bracket get revised and re-evaluated. Both methods anchor difficulty to demonstrated solver behavior, not human estimates, which historically correlate poorly with what models at a given level can learn.

CalibForge dual-mode calibration pipeline: candidate tasks are filtered and revised until solver behavior confirms learnability
FIG. 03 CalibForge dual-mode calibration pipeline: candidate tasks are filtered and revised until solver behavior confirms learnability — CalibForge paper (Renmin University of China)

Terminal-Bench 2.0 provides context: 89 Docker tasks spanning software engineering, scientific computing, cybersecurity, and ML. Frontier models top out below 65%. Smaller models cluster around 15%. CalibForge-trained models reach 32.58% and 47.57% depending on base model—solid gains given the ceiling, though still below frontier.

Terminal-Bench 2.0 pass rates by model tier: frontier models top out below 65%, smaller models cluster near 15%; CalibForge-trained models reach 32.58% and 47.57%
FIG. 04 Terminal-Bench 2.0 pass rates by model tier: frontier models top out below 65%, smaller models cluster near 15%; CalibForge-trained models reach 32.58% and 47.57% — CalibForge paper; Terminal-Bench 2.0

The stronger signal is out-of-distribution transfer. SWE-bench Pro and Doc2Repo were not part of training, yet models improved 27.68 and 30.04 percentage points. The calibration process selects for general terminal-agent capability rather than Terminal-Bench pattern-matching. Solver-relative learnability transfers across domains.

The paper omits compute cost for full calibration and heterogeneous pool composition details. Teams replicating this must build their own solver pool, making quality partly dependent on solver choice—a circular dependency if those solvers are also being trained.

For ML teams hand-authoring eval suites or using single-solver filtering, CalibForge offers a concrete alternative: replace human difficulty estimation with solver disagreement. The 5,431-task corpus is public. Solver-calibrated curriculum generation transfers, and the numbers suggest the method is ready to test on your stack.