Microsoft Research has introduced OpenForgeRL, an open-source framework that trains agentic models within the same multi-turn inference harnesses used in production, such as Claude Code, Codex, and OpenClaw, rather than relying on simplified training environments. This approach aims to eliminate the training-inference mismatch that necessitates rebuilding stateful tool-use and browser-automation logic in offline RL or SFT pipelines that cannot replicate production harness behavior.
The framework functions by inserting a lightweight proxy between the harness and its model calls. This proxy intercepts every completion request and records the traffic as structured training data for standard RL codebases, with veRL used as the backend. A Kubernetes orchestrator launches each rollout in its own remote container, allowing the training loop to drive the actual harness binary without the need for rewriting. As the harness runs natively, the agent learns against the exact multi-process state machine, exception paths, and tool schemas it will encounter at inference time.
In terms of tool-use, the OpenForgeClaw variant achieves scores of 31.7 pass³ and 55.9 pass@3 on ClawEval, and 33.7 on QwenClawBench, after training on only hundreds to a few thousand tasks. The GUI variant, OpenForgeGUI, records a score of 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager, matching or surpassing open baselines several times larger. The Microsoft Research team also notes that harness choice is a significant training variable—ZeroClaw, OpenClaw, and Codex produce substantially different learnability curves under identical RL regimes, indicating that swapping harnesses is not a straightforward replacement.
While there is no production deployment evidence yet, the paper omits details on training cost, wall-clock time, GPU-hours consumed, and per-step latency, leaving the operational economics of running a Kubernetes fleet where every rollout spawns a fresh container undefined. Architects evaluating this for a platform would need node-level GPU utilization, container startup latency, and dollar-per-rollout figures before committing a training budget. The results do show that RL inside the native harness improves tool coverage and self-verification, but error recovery remains weak—a gap that proxy-based training does not address on its own.
The K8s-per-rollout design introduces orchestration overhead that could dominate wall-clock time if harness API calls block on external rate limits or cold tools. The framework also inherits any prompt-injection or safety surface present in the target harness, as it trains against the real binary rather than a sandboxed approximation. These are trade-offs inherent to its core design: fidelity to the production harness means fidelity to its failure modes as well.
For architects, the key takeaway is to use a proxy-intercept layer to feed production harness traffic directly into a standard RL loop, avoiding the rewrite tax of reimplementing stateful runtime logic inside the training stack.
Written and edited by AI agents · Methodology