Vals AI test finds agent teams cost up to 5.1x more with minimal gains

Teams cost more but rarely score better

Vals AI tested GPT-6 Sol and Claude Opus 5.5 on the Vibe Code Bench, both as single agents and as teams, at medium reasoning effort and at maximum reasoning effort. The team setups cost between 1.8x and 5.1x more than the single agent runs.

Across four comparisons between teams and solo agents, only one result showed a statistically significant improvement. That case was GPT-6 Sol at medium reasoning, where the team scored 7.3 points higher than the solo agent. At maximum reasoning, the team setup gave neither Sol nor Opus 5.5 a real advantage.

Scaling buys speed before quality

Vals AI reported that its benchmark showed cost per app versus score on the Vibe Code Bench, with arrows pointing from each model single agent to its team at the same reasoning effort. The teams cost far more while barely improving scores.

Anthropic also saw quality gains shrink as it added more agents in two of its own tests with Opus 5.5. Larger teams reached a given performance level faster, but going from ten to 100 agents only nudged scores up slightly after 24 hours. In separate ProgramBench tests, speed gains came with higher token usage. On the knowledge base task, Opus 5.5 scored 0.53 with one agent, 0.70 with ten, 0.71 with 30, and 0.74 with 100. On Lean theorem proving, it scored 0.39 with one agent, 0.66 with ten, 0.66 with 30, and 0.68 with 100.

Fable 5.1 showed stronger quality gains on the Lean theorem proving task above ten agents, but it still scored below Opus 5.5 across all tests. On the knowledge base task, Fable score actually dipped slightly when scaling from 30 to 100 agents.

Coordination limits agent swarms

OpenAI researcher Noam Brown said on the Dworkesh Podcast that multi agent systems mainly buy speed rather than better quality. He said four agents solved tasks twice as fast but also cost twice as much, and that at 16 agents the pattern held while growing slightly less efficient. He said the effect depends heavily on the task, noting that web research and math parallelize well, but writing a novel does not. Brown added that using 10,000 agents on a novel would be just as pointless as using 10,000 people, and that scaling to very large numbers of agents remains largely unexplored because the costs are too high.

OpenAI developer Eric Provencher warned against using agent swarms for the same reason. He argued they are most likely wasted money because coordination between agents breaks down, and he called this problem the coordination tax.

Sources: The Decoder