LangChain released Tuned Evaluators, a managed evaluation service that automatically attaches quality feedback to production agent traces without manual prompt tuning or frontier-model costs. The first evaluator, Perceived Error, detects conversations where an agent misunderstood a request, made a mistake, or took the interaction in the wrong direction—either from explicit signals (user corrections, rejections) or inferred ones (contradictions, unresolved outcomes). The evaluator is available now for Plus and Cloud Enterprise LangSmith plans in the US.
Perceived Error uses a LangChain-specialized model post-trained on labeled conversational traces that outperformed every frontier model in benchmarking while reducing cost by up to 82% (98% in some early-partner workloads). Unlike frontier model-as-judge approaches, teams no longer need to write prompts, select LLM judges, or manage inference infrastructure—LangChain handles versioning, benchmarking, and credential management end-to-end. A thread becomes eligible for evaluation after at least two human-AI message pairs and an idle period, with results attached within 12 hours.
For agent builders: quality evaluation at production scale is a blocking lever for agent improvement. Most users never submit explicit ratings, so Perceived Error (inferring errors from conversational signals) is a higher-fidelity proxy than user stars. The 82% cost reduction vs frontier judges makes continuous evaluation viable for cost-conscious teams; Vanta (early partner) describes it as a "safety net to catch failure modes from day one" while building domain-specific evals. This pattern—turnkey evaluators + team-specific fine-tuning—is becoming standard agent-development workflow.