LangChain shows agent routing cuts cost 74% with six-point accuracy loss

LangChain published an empirical routing benchmark showing 7% of agent turns drive 68% of total model spend—and routing those turns to the right model cuts cost by 74% while dropping accuracy six points. The benchmark used NVIDIA NeMo Switchyard, an open-source routing library, to split calls between Claude Opus 4.8 and NVIDIA Nemotron 3.5 Lightning, a 30B open-weight model.

The evaluation suite ran 145 multi-step agentic tasks averaging 6.3 model calls each, drawn from τ²-bench airline, Berkeley Function Calling Leaderboard, FRAMES, and Nexus. Workload types included customer support dialogue under policy constraints, on-call incident investigation, and multi-step workflow automation across messaging, issue tracking, and email. All costs use cached-input pricing.

The routing approach LangChain benchmarked uses Switchyard's LLM classifier in escalation mode. Each task starts on Nemotron 3.5 Lightning. A judge model evaluates each turn and votes on agent quality. Two consecutive negative verdicts trigger one-way escalation to Opus 4.8 for the remainder of that session. The one-way door matters: it stops the judge from firing on post-escalation turns, keeping judge overhead bounded.

Running Opus 4.8 alone costs $11.45 per run and $0.092 per task at 86% accuracy. The routed arm—Nemotron handling 93% of calls, Opus handling 7%—costs $3.00 per run and $0.026 per task at 80% accuracy. Nemotron alone costs $0.72 per run at 77.7% accuracy. In the routed arm the judge model consumed 21.2% of total spend, making it the second-largest line item after Opus. The judge gets no benefit from prompt caching and fires on every turn until escalation, accumulating spend across the 93% of turns that never escalate.

LangChain provides a break-even formula: divide judge cost by the price gap between your two models. That ratio is the minimum fraction of calls that must route to the cheap model for routing to pay off. When the two models are close in price, the ratio can exceed 100%—meaning routing cannot recoup its overhead unless the cheap model is self-hosted. That boundary makes model selection upstream of routing configuration a math decision before buying anything.

The authors flag a scoping constraint directly. The Deep Agents suite is controlled, and that control saturated it: accuracy variance between the 30B model and the frontier model was only 8 points, giving routing less room to show value than a harder or more open-ended workload would. The 6-point accuracy gap between routed and Opus-only arms shrinks in proportion to how close the two models perform on your actual task distribution. Whether the gap widens or narrows on production traffic is something the benchmark cannot predict.

Switchyard ships as either a proxy your agent points at or as middleware embedded in the agent process. Beyond the LLM classifier it includes a stage router that reads error patterns, reasoning patterns, and token counts with no extra model call, and a prefill-activation MLP that routes on internal model activations—the latter still in research. LangChain benchmarked only the LLM classifier in escalation mode.

Run the break-even formula against your model pricing before configuring a router. Then measure your task distribution's accuracy gap before committing to the six-point loss.