LangChain cuts evaluation cost 100x with a fine-tuned open model

LangChain and Fireworks published a trace-judging system that cuts per-trace evaluation cost by 100x compared to calling frontier APIs directly. The system fine-tunes Qwen-3.5-35B to classify "perceived error"—whether a user believed an agent made a mistake—on every production trace flowing through LangSmith, which processes billions of tokens daily. Frontier model pricing made comprehensive eval economically impossible at that scale; the fine-tuned open model makes it routine.

Perceived error means not whether an agent was objectively wrong, but whether the user responded as if it was. Indicators: user corrections, repeated requests, rejected agent actions, explicit assistant admissions of mistakes. The evaluator outputs a JSON object with a boolean flag and rationale—{"perceived_error": true, "reason": "The user corrects the meeting date the assistant used."}—and writes it back into the trace for downstream analysis.

Building the training set avoided pure human annotation at scale. LangChain sourced multi-turn traces from two internal datasets: chat-langchain (885 examples, technical Q&A) and Fleet (911 examples, a no-code document and research agent). Labels came from panel-of-models voting; disagreements escalated to a second panel; persistent disagreements went to human reviewers. Perceived-error rates were 24% and 18% respectively—real production distribution, not artificially balanced splits.

Fine-tuning used managed supervised fine-tuning with LoRA on Fireworks, training exclusively on chat-langchain. The goal was confirming cross-domain transfer before committing to dual-dataset training. The chat-langchain SFT model hit 96.1% accuracy on its home dataset versus 91.6% for Claude Opus and 90.5% for base Qwen. On Fleet—a domain it had never seen—the SFT model scored 90.8%, 3.6 points above base Qwen and 0.6 above Claude Opus at 90.2%. GPT-5.5 led on chat-langchain at 98.9% but dropped to 89.1% on Fleet, below the fine-tuned open model.

Qwen-3.5-35B was chosen after smaller models failed to handle multi-turn trace reasoning. The team excluded tool-call messages from training context, betting that human and AI messages carry the primary perceived-error signal. Prompt engineering followed small-scale failure-mode analysis rather than wholesale rewriting. Both choices are marked as levers for future work—reintroducing tool calls, trimming long sequences—but the current setup already clears frontier accuracy on cross-dataset transfer tests.

Running a 35B open-weight model on Fireworks inference costs a fraction of frontier API calls per token. At LangSmith's scale of billions of tokens daily, that difference is binary: comprehensive trace evaluation is either affordable or impossible. The 100x reduction moves it into the first category. Teams with smaller agentic workloads get the same benefit—eval coverage that was previously sampled can now span full production traffic.

The main friction: this approach requires a labeled dataset and a fine-tuning cycle. LangChain built theirs from production traces using model-assisted labeling, which is replicable but not trivial. Any team wanting a domain-specific perceived-error judge will need to run the same process on their own traces.

If eval budget constrains trace-judge coverage, fine-tuning a mid-size open model on Fireworks is now a credible path to frontier-quality signal at 1% of frontier API cost.