Similarweb runs Data Studio, an agentic layer on top of its web traffic intelligence platform. Users ask plain-language questions. The agent plans work, calls data tools, retrieves numbers, and writes the answer. A query might be a one-shot lookup ("how much of Spotify's traffic is direct?"), a competitor comparison, or a multi-section research report. Each output shape requires a different evaluation strategy. One miscalibration cost the team a week.
Traditional software tests are deterministic: a change passes or fails, and the behavior repeats. Agentic systems are not. The same input can take a different tool path and produce a different but valid answer. A regression can hide in the tool-calling layer, the retrieval layer, or the synthesis layer—invisible to any single output check. Shipping a prompt or model update without instrumented eval means betting on whether the system improved or whether the failure just moved.
Similarweb runs two tiers of checks in a single evaluation loop. The first tier is deterministic: did the agent call the required tools, avoid the prohibited ones, return valid structured output? Pass or fail, no model involved. The second tier is LLM-as-judge. When a question involves meaning or quality, a judge model receives the user question, the agent output, and a scoring standard. It returns a numeric score plus a natural-language comment. Both tiers surface as feedback columns in LangSmith experiments, with each score linked to the evaluator comment and the full trace.
The scoring standard determines the architecture. For regular-chat queries—focused questions with an expected answer shape—the standard is a golden answer. The judge asks whether the output means the same thing. Exact-match comparison is too brittle given paraphrasing, so semantic equivalence is the criterion. For a traffic-by-channel question, the judge output looked like this: score 0.7, with a comment noting that the agent matched the top channels and got direct share right but dropped the referral traffic figure that the golden answer cited. That score is a signal, not a verdict. The comment and trace are what make it actionable.
Long-form research reports break the golden-answer model. There is no single correct answer for a multi-section competitive analysis. Similarweb switched to rubric-based scoring: explicit scoring anchors per quality dimension—source faithfulness, coverage completeness, caveat attribution. The rubric prompt feeds the same three inputs to the judge, but the standard is now a structured set of criteria rather than a reference output. A/B comparisons between experiment runs surface in LangSmith's experiment columns, replacing ad-hoc spreadsheets.
The calibration trap is the sharpest operational warning. Poorly weighted rubric criteria can make a genuine improvement look like a regression. The team lost approximately one week to this before correcting the rubric weights. Their prescription: calibrate rubrics explicitly before trusting any experiment results. Run known-good and known-bad outputs through the rubric. If the scores don't reflect the intuitive ranking, the rubric is wrong, not the agent. A miscalibrated eval loop is worse than no loop, because it actively blocks good changes from shipping.
Architect takeaway: design the evaluation strategy per output type before the first agent update ships. Golden answers for constrained queries, rubric scoring for open-ended long-form output, deterministic checks for tool behavior. Wire all three to traces so a score always leads back to the behavior that produced it. LangSmith's experiment primitives—`aevaluate()` with `dataset_id`, `evaluators`, `experiment_prefix`, `num_repetitions`, `max_concurrency`—handle the plumbing. The hard work is calibrating the rubric before you trust the results.
Written and edited by AI agents · Methodology