LangChain shipped Tuned Evaluators in LangSmith on August 18, 2026 — a managed, post-trained judge that attaches quality feedback to every production conversation without requiring teams to build, label, or maintain an evaluator. The first release targets Perceived Error: detection of conversations where an agent made a mistake, misunderstood a request, or took the interaction in the wrong direction.
Teams attach a Tuned Evaluator to an existing LangSmith tracing project. LangSmith identifies eligible threads — those containing at least two human-AI message pairs that have reached the configured idle period — and runs the judge against each one. Results and explanations attach to the trace as structured feedback. Teams then filter to flagged conversations, compare repeated failure modes, route ambiguous threads to human review, or pipe tagged examples into evaluation datasets. LangChain owns the evaluator prompt, judge model selection, versioning, and inference infrastructure.
The judge flags user corrections, repeated requests, and rejected actions. It also infers problems from contradictory responses, acknowledged mistakes, persistent misunderstandings, and unresolved outcomes — failure patterns that never surface in thumbs-up/thumbs-down ratings.
The 82% cost reduction comes from frontier model calls. Some early-partner workloads reached 98% savings. LangChain says the specialized post-trained model exceeded every frontier model in their internal benchmark. Evaluation completes within 12 hours of a thread becoming eligible. For teams that restrict frontier-model evaluation to a fraction of production traffic because full coverage is too expensive, this changes the operating envelope: you get signal on every conversation, not a spot-checked slice.
Vanta was an early partner. Kevin Royer, Staff ML/AI Engineer at Vanta, called the evaluator "a safety net to catch failure modes from day one, while we built out our own business-specific evaluators." Tuned Evaluators serve as baseline coverage a team stands up immediately while writing evaluators that reflect their product's quality criteria.
Building the evaluator has always required resolving ambiguous labeling cases, producing labeled data to calibrate a judge, benchmarking model choices, and maintaining all of it as production behavior drifted. Most teams skipped this work and ran frontier models on a sample. Tuned Evaluators collapse build time to zero for the Perceived Error case. The tradeoff: you run LangChain's post-trained model, not your own prompt you can inspect and tune. Eligibility criteria and evaluation windows are also fixed — threads under two human-AI pairs don't qualify, and results lag up to 12 hours.
| Dimension | Build Your Own | Tuned Evaluators |
|---|---|---|
| Build time | Significant — labeling, benchmarking, tuning required | Zero |
| Labeled data required | Yes | No |
| Inference cost | Frontier model rates on sampled traffic | ~82% cheaper; up to 98% for some workloads |
| Coverage | Spot-checked sample (cost-limited) | Every eligible conversation |
| Model choice | Team selects and benchmarks | LangChain's post-trained model |
| Prompt inspection & tuning | Full access | Not available |
| Eligibility criteria | Flexible | Fixed: ≥ 2 human-AI pairs + idle period |
| Result latency | Variable | Up to 12 hours post-eligibility |
| Maintenance | Team-owned | LangChain-managed (versioning, drift) |
The Perceived Error evaluator is available now on LangSmith Plus and Cloud Enterprise plans in the US. LangChain has not disclosed what additional Tuned Evaluators are in the pipeline or whether the managed judge will be exposed for fine-tuning on team-specific labeled data.
If your agent ships without systematic Perceived Error coverage because building the evaluator felt expensive, attach this immediately. 82% cost savings over frontier judges at full trace coverage is the reason to do it today.