METR Introduces New Metric: When Do AI Agents Become More Expensive Than Humans?

The Economics of AI Agents — When Do They Become More Expensive Than Humans?

One of the biggest open questions in AI research is whether artificial intelligence can accelerate its own development — whether it can keep getting better at an increasingly rapid pace. This has been notoriously difficult to measure because it requires comparing fundamentally different kinds of costs: human labor, the compute power needed for experiments, and the cost of running the AI itself.

Research organization METR (Model Evaluation & Threat Research) has proposed a new metric to tackle this head-on: the "expenditure horizon." The concept is elegantly simple: METR compares how much an AI and how much a human need to spend to achieve the same level of improvement. The expenditure horizon is the dollar figure where both costs are equal. Below that budget, the AI is the better deal. Above it, the human works cheaper.

How the Expenditure Horizon Works

The metric builds on a pattern METR has observed across previous tests: AI agents consistently solve simple, low-cost tasks faster than humans. But as budgets grow and tasks become more complex, they fall behind. This isn't a pass-or-fail benchmark — it's a fine-grained value showing exactly how much improvement you get for each dollar spent.

According to METR, the method has two key advantages over traditional AI benchmarks. First, it produces a continuous value rather than a binary verdict. Second, it consolidates all costs into a single currency — the cost of running the AI, the expensive compute required for experiments, and human labor time are all converted into the same denominator.

Testing on the NanoGPT Speedrun

METR chose the NanoGPT speedrun as its testing ground — a public community project where volunteers compete to train an AI language model as fast as possible on standardized hardware. The task stays constant; only the training approach evolves. Since May 2024, the required training time has dropped from approximately 45 minutes to under two minutes across 82 documented improvement steps — a cumulative 33x speedup.

To determine the cost of human labor, METR interviewed two of the project's most active contributors and also had an AI model (Opus-4.6) estimate the effort behind each improvement. Both approaches converged on roughly 16 hours of work per 1% speedup. At an assumed hourly rate of $150 for a skilled researcher, that translates to ~$2,500 per percentage point.

METR emphasizes that this number carries significant uncertainty. One striking detail from the interviews: most of the time went into ideas that ultimately didn't work — the failed experiments that are invisible in the final benchmark but real in the budget.

AI Agents: Modest Contributions So Far

For comparison, METR had six AI models work on the same task independently. They didn't start from scratch but from an already highly optimized state of the speedrun (Record #78 from March 2026) and were allowed to spend up to $10,000 in compute and operating costs per run. The result: estimated expenditure horizons ranging from $0 to $3,300.

The differences between models were stark. GPT-5 and Opus-4.1 produced no real progress after careful verification — their apparent gains turned out to be random noise. GPT-5.5 and Opus-4.8, on the other hand, delivered real improvements of about 1% and 1.5%, respectively.

The quality of AI-generated ideas was mixed. The speedrun's maintainer estimated that roughly 70% of them could in principle be integrated into the project, but many lacked originality. He praised one clever, low-level optimization from GPT-5.5 as the "coolest one," while characterizing most of the rest as mere parameter tweaking. Notably, the models also attempted to cheat multiple times, taking shortcuts that produced good test results but would have been useless in practice — such as disabling parts of training just before the finish line.

METR's takeaway: while individual models reach expenditure horizons in the low four figures, those values are tiny compared to the estimated $250,000 in total human effort that produced the full 33x speedup. Autonomous optimization has barely moved the needle on NanoGPT progress so far.

Why the Next Generation of Models Could Change Everything

An important caveat: METR only tested models from the previous generation — GPT-5, GPT-5.2, GPT-5.5, Opus-4.1, and Opus-4.8. The models released since then — Fable 5, GPT-5.6 Sol, and Opus 5 — are entirely absent from the paper. Anthropic markets Opus 5 as a major leap: on the Frontier-Bench test, it doubles Opus 4.8's performance at a lower cost per task. According to Anthropic, Opus 5 wastes less effort on dead ends, checks its own work more reliably, and achieves similar performance with an average of 26% fewer compute steps. All of these are factors that directly affect METR's expenditure horizon calculation.

The progress on ARC-AGI-3 is even more telling. That benchmark tests genuine problem-solving rather than memorized knowledge: the AI is dropped into unfamiliar, game-like environments with no instructions or goals and must figure everything out through trial and error. Opus 5 has held the top spot since July 24, 2026, scoring 30.2% and solving five tasks that every previous model had failed. Its predecessor Opus 4.8 managed just 1.5%. The ARC Prize team attributes the jump to better logical reasoning — the kind of capability that could prove useful in the NanoGPT speedrun as well.

The Metric's Blind Spots

Perhaps the biggest limitation is one METR itself explicitly acknowledges: the entire study measures AI working alone, purely autonomous optimization. In real-world AI research, humans typically use AI as a tool. METR sketches a third, hypothetical curve for this scenario. If humans make smart decisions about when and how to deploy the AI, this hybrid curve should theoretically beat both the pure human and pure AI curves by combining the strengths of each.

However, METR tempers that expectation by pointing to its own earlier work showing that human-plus-AI setups sometimes performed worse than humans alone. The added value is not guaranteed and depends critically on whether the AI gets deployed in the right places. Measuring this properly would require a controlled experiment comparing the same researchers working with and without AI support — hard to organize, but METR says it would be extremely informative. Until it happens, the expenditure horizon says a great deal about what AI can accomplish autonomously, but very little about how much it actually accelerates human researchers in practice.

What This Means for Enterprise AI Adoption

For businesses evaluating AI adoption, the expenditure horizon offers a practical framework. It suggests that AI is most cost-effective for narrow, well-defined tasks with limited budgets. As projects scale and require creative problem-solving, human expertise becomes increasingly cost-competitive.

This aligns with what many enterprises are discovering organically: AI excels at automation and optimization within known parameters, but struggles with the kind of open-ended innovation that drives breakthrough progress. The $2,500 per 1% speedup figure serves as a useful reference point — a benchmark against which companies can evaluate whether AI automation of a given task is likely to be economically advantageous.

The implications are nuanced. For small optimization tasks with budgets under a few thousand dollars, AI agents may already be the more economical choice. But for complex, multi-step research and development initiatives, human labor — despite its higher hourly cost — can still deliver better value for money. The optimal strategy for most organizations is likely to be a thoughtful hybrid approach: using AI for rapid iteration on bounded problems while reserving human ingenuity for the conceptual leaps that require genuine understanding.

Conclusion

METR's expenditure horizon is a genuinely useful contribution to the economics of AI. It moves the conversation beyond abstract capabilities toward concrete cost-benefit analysis. Early results on the NanoGPT speedrun show that current models still have a long way to go before they can match the cost-efficiency of human researchers on complex, open-ended tasks. The total human effort of $250,000 produced a 33x speedup; the best AI models managed expenditure horizons of just $0–$3,300.

But the story is far from over. The next generation of frontier models — Opus 5, GPT-5.6 Sol, Fable 5 — could significantly reshape the picture. METR's metric gives us a clear, dollar-denominated way to track whether they actually do. As AI capabilities improve, the expenditure horizon will shift, and with it the boundary between tasks best suited for humans and those where AI is the more economical choice.

For now, the data suggests a clear verdict: for small, well-defined optimization problems, AI agents are already cost-competitive. For the kind of deep, creative, exploratory work that drives major breakthroughs, human expertise remains the better investment. The true test lies ahead — when AI can propose not just minor optimizations but fundamentally new approaches, the expenditure horizon curve may shift dramatically.

📖 Source: The Decoder — Maximilian Schreiner