LangChain shipped ReviewBench on July 31, 2026 — a 59-task evaluation harness built from real pull-request feedback in the LangSmith mono-repo. It is the first code-review eval grounded in curated reviewer findings rather than synthetic bugs, deployable today against any agent configuration that speaks Harbor task format.
Most existing code-review benchmarks measure wording similarity between generated comments and human-written ones, which tells you nothing about whether an agent caught the actual defect. ReviewBench starts from a different place. LangChain collected comments from trusted reviewers on merged PRs, passed the raw set through an LLM gate to filter nits and questions, then manually reviewed every remaining candidate. Only comments that identified a real issue introduced by the change and were specific enough for verification made the cut. That curation pipeline produced 59 tasks covering 64 baseline issues.
Each task is frozen: a local GitHub stub serves the PR metadata and diff so runs are deterministic and don't depend on live GitHub state. The agent receives the frozen context, can inspect the full seeded repository, then submits a structured findings list with location, title, and explanation. A hidden LLM-as-judge verifier compares the submission against the curated baseline. The headline metric is F1, combining coverage (did the agent find the baseline issue?) and precision (were submitted findings actually supported by the code?). Extra valid findings count for precision but do not inflate coverage.
Running multiple models through the same base Deep Agents harness produced a clear result: the strongest runs recovered roughly 30% of the curated baseline issues. Luna and Terra scored lower than expected. Their review strategy was narrow, stopping after a small number of findings. That kept token costs down but hurt coverage because many target issues require reasoning past the most obvious changed lines.
Two representative task types show why. One involves a database query that fetches and deletes a resource by ID without checking the tenant, requiring the agent to reconstruct an implicit project-level safety rule from surrounding code. Another involves an endpoint migration that silently dropped a filter present in the original API — catching it requires comparing both implementations for behavioral parity, not just scanning the diff.
A follow-up ablation on 20 tasks tested whether prompting alone could move the needle. A tuned Luna configuration with high reasoning effort, a structured review prompt, and no additional tools ran against the same 20 tasks alongside Opus 4.8 and Kimi K3. The prompt changes alone lifted Luna's results substantially, signaling that scaffolding design, not just model capability, is a live variable in code-review agent performance.
A parallel academic effort, c-CRAB from NUS and SonarSource, arrived at consistent conclusions through a different construction method. c-CRAB evaluated PR-agent, Devin, Claude Code, and Codex together. All existing agents combined solved only about 40% of c-CRAB tasks. Both datasets converge on the same finding: current review agents report valid issues at reasonable precision but systematically miss the specific, context-dependent findings that experienced reviewers catch.
For teams shipping code-review agents, ReviewBench fills a concrete gap. The tasks are hard in a representative way — they require implicit contract reasoning, cross-file comparison, and behavioral-parity checks that synthetic benchmarks tend to skip. The Harbor task format means you can run the harness locally, swap in your own agent, and get an F1 score against the same curated baseline LangChain used. At 30% recovery under minimal scaffolding, the ceiling is low enough that any improvement your system prompt or tool configuration produces will be visible and attributable.
If you are evaluating or shipping a code-review agent, ReviewBench is the first benchmark worth running — not because it covers every language or diff context, but because its 64 issues are real defects that real reviewers caught, scored against criteria that do not reward verbose output.
Written and edited by AI agents · Methodology