UK AI Security Institute Finds Standard Benchmarks Systematically Underestimate What AI Agents Can Actually Do
A UK AISI study across 7 benchmarks reveals that fixed compute budgets systematically underreport AI agent capabilities — increasing token budgets tenfold boosted success rates by ~25% on software engineering tasks.
🔍 Why Standard Benchmarks Fail to Measure True AI Agent Power
The UK AI Security Institute (AISI) published a study that has become one of the most important findings in AI evaluation. The institute tested leading frontier models on seven different benchmarks and discovered that standard tests systematically underestimate AI agent capabilities.
The reason is simple: every benchmark has a fixed compute budget — the maximum number of tokens an AI agent can use. AISI researchers proved that an AI agent's performance is not a fixed point, but a curve that grows with test-time compute. When we cut the budget while this curve is still rising, the resulting score shows the minimum, not the maximum. This discovery fundamentally changes how we should think about AI evaluation.
📊 The Compute Budget — The Hidden Variable
Imagine an AI agent trying to solve a complex software engineering task. The agent needs to think through the problem, generate multiple approaches, test them, debug, and iterate. Each step consumes tokens. A standard benchmark might allow only one cycle of this process. But what if the agent needed 5 cycles? AISI's research shows that when the compute budget was increased tenfold, success rates on software engineering tasks jumped by approximately 25%.
This creates a "ceiling effect" — the benchmark measures the test, not the agent. Just as an athlete timed with a broken stopwatch will always appear slower, AI agents limited by insufficient compute budgets appear less capable than they actually are.
🔬 HealthBench — When More Compute Doesn't Help
Interestingly, AISI found that not all benchmarks behave the same way. On HealthBench (medical knowledge), increasing compute did not improve scores. This is because some tasks are "recognition" based (you either know the information or you don't) — no amount of additional thinking will help if the knowledge simply isn't in the model's training data.
This distinction is critical: benchmarks that test pure knowledge produce stable, fair results. But benchmarks that test reasoning, problem-solving, and agentic behavior — such as SWE-bench, GAIA, and CyberSecEval — are actually measuring a moving target constrained by artificial token limits. The gap between reported scores and real capability widens as models get better at reasoning.
📈 Human Time vs AI Tokens — A Power Law Relationship
AISI discovered a near-perfect power law relationship: doubling the compute budget produces a consistent ~15% improvement across software engineering and cybersecurity benchmarks. This relationship is consistent across different model families — from OpenAI's GPT series to Anthropic's Claude to open-source models.
This means that any benchmark reporting a "single score" for an AI agent is fundamentally misleading. The score does not reflect the agent's capability ceiling but rather the arbitrary budget ceiling set by the test designers. For frontier AI evaluation, this calls for a paradigm shift: instead of reporting one score, researchers should report compute-performance curves — showing how performance improves as more compute is allocated.
🔮 Newer Models Get Better Returns on Compute
Perhaps the most important finding: newer models show steeper compute-performance curves than older ones. This means the gap between reported benchmark scores and real capability is widening over time. As models become more efficient reasoners, they benefit more from additional compute — but standard benchmarks haven't adjusted their budgets accordingly.
For the industry, this creates a dangerous blind spot. Regulators, researchers, and businesses making decisions based on published benchmark scores may be operating with a systematically incomplete picture of what frontier AI can actually do. A model that scores 60% on a standard benchmark with a tight compute budget might achieve 85% given sufficient compute — a difference that could be critical for safety assessments or deployment decisions.
⚠️ Why This Matters for AI Safety
This finding has direct implications for AI safety. If international AI safety institutes and regulators rely on current benchmarks, they may underestimate the real-world capabilities — and risks — of frontier AI models. The gap between apparent and actual capability is largest precisely in the areas that matter most for safety: cybersecurity, autonomous reasoning, and long-horizon planning.
For Georgian businesses and developers, this research offers an important lesson: when evaluating AI models for your projects, don't rely solely on published benchmark scores. Test the model yourself with your specific use case and allot sufficient compute time. The model that scores lower on a benchmark might outperform others when given adequate resources — especially in complex, multi-step tasks typical of real-world business applications.