A third of code patches that pass all other tests fail end-to-end serving validation on SGLang tasks, revealing that local task completion does not guarantee production correctness. Researchers at NVIDIA and UC Berkeley built SWE-Serve, a 53-task benchmark grounded in production changes to SGLang, an open-source inference-serving system, to measure this gap directly.
The benchmark spans six inference engineering families: model and backend enablement, speculative and advanced decoding, kernels and quantization, caching and runtime state, distributed execution, and serving APIs. Tasks range from localized correctness fixes to integrations crossing API, scheduling, and runtime-state boundaries. The median oracle solution changes 553 lines across seven files, with some tasks requiring up to 6,077 lines across 35 files. Each task executes on CPU or a single H100 GPU and includes hidden functional and regression tests, with 19 tasks adding model-serving end-to-end tests that launch a standalone serving process and validate behavior through public request interfaces.
Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1. Claude Opus 5 and GPT-5.6 Sol both reach 75% pass@1, though at different resource costs: Opus 5 at maximum effort costs $17.40 per task with 122k output tokens, while Sol at maximum effort costs $12.26 with 52k tokens. The benchmark reveals a 40-percentage-point spread in performance across models, from 75% for the top performers to 35% for Inkling S.
The production correctness gap emerges starkly when model-serving end-to-end tests are removed from scoring. On 19 tasks with end-to-end coverage, pass rate jumps from 45.9% under the full verifier to 69.4% when E2E tests are excluded, a 23.4-percentage-point increase that holds across all 11 top-per-model configurations. A matched control removing the same number of non-E2E tests produces only 8.0 fail-to-pass transitions on average, compared with 16.1 for E2E test removal—a 2.0× difference. On the Gemma 4 mixture-of-experts core-serving task, 48.5% of agent-created patches passed every non-E2E test but failed at least one E2E test, unable to serve the specified model correctly through the live server's public interfaces.
Beyond end-to-end serving, tasks testing multiple runtime domains show substantially lower pass rates. The 26 single-runtime-domain tasks average 69.0% pass rate, while the 27 multi-runtime-domain tasks average 47.7%—a 21.3-percentage-point gap. Tasks explicitly testing concurrent coordination show even steeper drops, with pass rates 29.0 percentage points lower than tasks without that requirement. Persistent-state requirements correlate with 20.8-percentage-point lower pass rates on GPU tasks. These gaps persist even at maximum reasoning effort: increasing effort improves aggregate pass rates but does not consistently close the production correctness gaps.
The benchmark enforces closed-book evaluation, blocking public web and upstream repository access during agent execution. An open-book pilot confirmed that all three models evaluated retrieved task-specific upstream solutions, so the closed-book setting prevents agents from copying solutions. Every evaluation trajectory was audited for prohibited retrieval; no trials were invalidated on this basis. The benchmark admits only 34% of task candidates after qualification: 53 of 156 candidates passed executable controls requiring no-op patches to fail all fail-to-pass tests and pass all pass-to-pass tests, while oracle solutions pass all tests. Agent-assisted adversarial probing identified vulnerabilities in task instructions and verifiers, leading to 27 task candidates requiring verifier changes before admission.
For teams shipping inference systems, the takeaway is direct: local test passage does not validate production serving, and end-to-end tests through live serving paths must be part of the evaluation gate.