AI coding agents still need human cleanup on nearly three out of four tasks in real-world test

Researchers from Alibaba and several Asian universities created Co-Design Bench, a coding-agent test based on real user change requests for existing web apps. Even the best of ten models tested, GPT-6 Astra, fully completes only about one in four tasks, while meeting about 70 percent of requirements on average; in most cases a user must still clean up at least one detail by hand.

Why older benchmarks look too easy

The authors say existing benchmarks have become saturated. When coding agents can run, check, and fix their own code, they score above 99 percent on those tests, and differences between models can barely be measured. On older benchmarks such as WebCompass-edit, agentic coding models reach near-perfect scores, while Co-Design Bench leaves a gap of more than 35 percentage points.

How the new benchmark works

Co-Design Bench draws on real requests from actual coding sessions. The researchers favored cases where users had to rephrase a request multiple times or roll back changes, so the test mainly contains tasks that already gave AI agents trouble in practice. Tasks range from small tweaks, such as pulling an amount from a receipt and updating every dependent total, to full-scale project overhauls. In the multimodal track, agents must extract values from receipts, tables, or videos and update all dependent views consistently. Grading is strict: one AI judge tests the finished app in a browser, then a second re-checks every positive verdict and looks for mock-ups or hardcoded results. Points go only to features that actually work in the app.

Models struggle with combined changes

Behind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max take the next spots, while Grok 4.6 lands in last place. On larger overhauls, the much smaller Qwen3.8-27B climbs to second, but the authors do not derive a general advantage for that model from the result. Performance drops across all models as a task combines more types of changes, including layout, design, interaction, and content. Every one of the ten models scores worse once a task demands three or four different types of changes. Extra compute does little to fix this weakness: GLM-5.2 burns through nearly six times as many tokens as GPT-6 Astra on smaller changes and still scores significantly worse. GPT-6 Astra leads both tracks but fully solves only 41 percent of multimodal tasks and just under 18 percent of repository tasks.

Harness and test limits

In a separate test, the researchers ran the same models in different harnesses, the programs through which an agent reads, writes, and executes code. Qwen3.8-27B scored 75 or just 56 percent on larger overhauls depending on the environment, and which of two models came out ahead also depended on the environment. Leaderboards that list only the model therefore tell only part of the story, and the authors recommend always reporting model and working environment together. For companies choosing a coding agent, that means testing both components as a pair. The benchmark also has limits: it evaluates one fixed request per task, so follow-up questions or mid-task corrections are not part of the test, and the AI judges can make mistakes. A study by METR found that benchmark results and real-world usability can diverge, with the actual acceptance rate of AI-generated code by project leads roughly 24 percentage points below scores on SWE-bench Verified. Flaws in test environments can also let agents pass tasks without actually solving them. The effect of the working environment is not clear-cut, though: a recent study found barely any measurable benefit from more elaborate harnesses, even though they consumed up to three times as many tokens.

Sources: The Decoder