Smevals cuts model eval cost by decoupling grading from test runs

GPT-5.5 and GPT-5.6 Sol cost twice what GPT-5.4 did. Claude Fable 5 costs twice what Opus 4.8 did. Even Gemini 3.5 Flash-Lite — Google's budget tier — is pricier than Gemini 3.1 Flash-Lite. Simon Willison and Prime Radiant released smevals on July 31 to answer the obvious question: before paying 2× per call, can you prove the cheaper model is already good enough? The tool is a Python CLI that runs structured eval suites across models, prompts, and agent harnesses, installed with `uv tool install smevals` or `uvx smevals --help`.

The core data model has five concepts: an eval (a named capability question), tasks (individual exercises), configs (model + parameters + optional harness), runs (immutable execution records), and grades (grader output — pass/fail plus numeric score). Configs can wrap bare model calls or full agent runtimes like Claude Code, Codex, or Pi; the runner is any executable that reads three environment variables (SMEVALS_MODEL, SMEVALS_PROMPT, SMEVALS_RUN_DIR) and writes output.txt. The grader is a YAML-configured sequence of checks — built-in operations like `contains` and `xml-valid`, or custom checker scripts that spawn LLM-as-judge calls. Checkers share a working directory, so a render-svg checker can produce a PNG artifact that a subsequent vision-model checker then evaluates.

One operational advantage: runs are immutable and grading is decoupled. You can re-run a different grader over existing runs without re-executing tasks against the model. This lets you collect outputs cheaply — say, gpt-4.1-mini at 5.0 and 12.6 seconds per task — iterate on your rubric, and re-score without extra API spend. The `-n 5` flag tops each task to five runs to handle non-determinism; harness-level failures stay on disk for debugging but are never graded and do not count toward the -n target.

The novel angle is agent scaffolding. Running `uvx smevals docs` dumps the bundled README into the agent's context. A single natural-language prompt — "build an eval that tests how well models can write haikus, with two tasks: a haiku about a pelican and a haiku about two otters in love" — produces a complete seven-file eval directory. The initial grader Codex generated checked only for three non-empty lines. A follow-up prompt — "improve the grader to check for the right consonants and vowels using gpt-5.5" — added a `checkers/haiku-judge` LLM-as-judge checker. The feedback loop is: agent scaffolds, human inspects grades, agent tightens the rubric. The eval infrastructure becomes something a coding agent maintains rather than something a human writes from scratch.

Compared to heavier eval frameworks — OpenAI Evals, Inspect, deepeval — smevals makes no assumptions about your inference stack. The runner is a shell script; any model accessible from the command line (local ollama endpoints, API providers, agent harnesses) works. The filesystem layout is the protocol: eval.yaml, tasks/, configs/, graders/, checkers/, runs/. Results render in the terminal or as static HTML for team sharing.

Two gaps worth noting: there is no built-in cost tracking per run (latency only, not dollar spend), and the grader contract assumes deterministic scoring logic. A non-deterministic LLM judge on a borderline case can produce inconsistent grades across decoupled re-grades. Both are tractable, but teams wiring smevals into CI pipelines should build those guardrails themselves.

Before the next model-tier upgrade discussion, ship a smevals suite against your actual task distribution, run it against gpt-4.1-mini and your current production model, and let the grades settle the argument rather than benchmarks someone else ran.