New Parsimony Benchmark Ranks AI Coding Agents by Code They Add

Existing AI coding benchmarks measure solve rate and cost but do not punish excess code, which can make a codebase harder to work with over time. Parsimony, a new benchmark, takes a different approach: it measures how much code AI models add when they solve a task.

How Parsimony works

Parsimony takes the patches that benchmarks already publish and counts coding units, meaning syntax elements such as statements, calls, comparisons, names and literals. Lines are not counted, so cramming code into fewer lines is not rewarded. Deleted code is subtracted, giving a simple formula: net units equals added minus removed. Each patch is applied to the task's code and both versions of every changed file are parsed; nothing is run. Tests, docs and generated files are left out, and only Python is measured. Pass and fail results come from SWE-bench's published results, and tests are not rerun.

Reading the leaderboard

Models are ranked by mean net units added on the tasks they solved, lowest first. The "Net units added (mean)" figure also includes failed attempts. The "Solved" column comes from SWE-bench Verified, and "Measured" shows how many solved attempts could be measured. The Artificial Analysis Intelligence Index appears as an outside score matched by model and reasoning effort, but it does not affect the rank. Faded entries on the board were run by the author on the first 3 to 10 DeepSWE Python tasks, using OpenAI models with pi and Anthropic models with Claude Code, and graded with each task's own tests; they are not verified or ranked. GPT-5.6 Sol was run both ways, and the author's run added 0.88 times the code of its published run.

Limits and caveats

The project notes that smaller is not always better: units measure size, not readability or design. Models that write more code also tend to solve more tasks, with Spearman correlations of 0.33 to 0.60 on the boards with 26 or more models. Every model uses the same agent, mini-SWE-agent, but the agent still affects how much code gets written.

Adding your own benchmark

Readers with code-size numbers from another benchmark can download an example JSON file, fill in their results and open it to get the same chart and table. The file stays in the browser.

Sources: parsimony.lasu.dev