NVIDIA's SWE-Serve exposes the gap between local tests and live serving

An AI coding agent's patch can pass its tests and still fail once a server loads a real model and starts handling requests. NVIDIA published SWE-Serve on September 23 to measure that gap. Built with input from the SGLang team, it turns 83 merged SGLang pull requests into 53 executable tasks.

What SWE-Serve tests

The tasks span six families: speculative and advanced decoding (14), model and backend enablement (12), kernels, quantization and performance (8), serving APIs and runtime correctness (8), caching and runtime state (7), distributed execution and scheduling (4). Twelve run on CPU and 41 on a single NVIDIA H100; the first release leaves out other inference engines, multi-GPU execution and multi-node serving. Nineteen tasks start a real server.

What live-serving checks catch

Across the 19 tasks that exercise a live server, the same 627 patches pass 45.9% of the time under the full verifier. Excluding the live-serving tests lifts that rate to 69.4%: 147 patches flipped from fail to pass, and roughly every third patch that cleared the other checks failed a serving test. The 19 tasks hold 276 serving tests, 242 of them taken or adapted from SGLang. The Gemma 4 MoE task shows the pattern plainly, with 16 of 33 patches passing every other check but failing at least one live-serving test.

NVIDIA stresses the narrow meaning of a pass: it says the patch satisfies the benchmark verifier, not that it is deployable, ready to merge or endorsed by SGLang maintainers.

Where agents score lower

The request-to-output path splits into four runtime domains: request handling and I/O, scheduling and the request lifecycle, model execution, and KV-cache and resource management. The 26 tasks confined to one domain pass 69.0% of the time; the 27 spanning several domains pass 47.7%, a gap of 21.3 percentage points that points the same way for every model setting.

Eleven models and 31 configurations ran under mini-swe-agent, a minimal Bash-only agent, in closed-book conditions. Mean pass@1 ranges from 34.6% to 75.5%. Claude Opus 5 and GPT-5.6 Sol lead at 75%, while four models tied at 64% differ sharply in cost, from $0.95 to $7.24 per task, and in mean wall time, from 25.5 to 99.9 minutes.

To build the benchmark, the team screened 786 potential sources, built 156 candidates and admitted 53; unmodified code had to fail the new-behavior tests while passing regressions. Evaluation blocks the public web and upstream repositories so that models cannot retrieve the upstream solutions.