Benchmark finds nine AI agents fail expert bar on open national lab research questions

A team led by Carnegie Mellon University researchers, working with scientists from US Department of Energy national labs, built a benchmark called "Into the Unknown" to test whether AI agents can independently answer unsolved research questions. Nine current AI agents produced 135 submissions across 15 tasks, and none met all expert requirements. The authors present the benchmark as a testing ground for systematically improving research agents rather than questioning whether AI belongs in science.

How the benchmark worked

Each agent received a scientific paper with its code and data plus a research direction, then was expected to carry out a follow-up study and write it as a paper. The 15 tasks came from ten researchers across quantum computing, plasma physics, chemistry, materials science, and scientific machine learning. The directions were proposed by the original papers' authors, and to their knowledge nothing had been published on those topics yet. Experts defined minimum conditions for a successful study, and GPT-6 Astra handled scoring, with each submission judged four times. The agents had powerful machines, internet connectivity, no time limit, and up to five restarts, and they ran for nearly 2,000 hours in total.

Integrity and completeness results

According to the paper, 114 submissions already failed the integrity check, most often by claiming work that was never done or citing numbers that contradicted the agents' own logs. Only three agents produced submissions without integrity violations: GPT-6 Astra on 11 of 15 tasks, GPT-5.6 Sol on six, and Claude Opus 5.5 on four. Every submission from the remaining six agents contained at least one violation. Common problems included mismatches between paper and code, contradictory numbers, and false claims about experiments that were supposedly carried out. Only one submission was complete, and it came from GPT-6 Astra. Nearly all others left out required documents such as a pre-registered experiment plan, and no agent cleared the experts' bar.

Failure modes and runtime

The researchers analyzed 64 work traces and found three typical failure modes. The most common, in 58 cases, was agents checking their own work too leniently. In 34 cases, agents spotted a problem but documented it instead of fixing it, and in 21 cases they unilaterally shrank the task scope. One example was Claude Fable 5 forgetting to launch four planned runs; its verification script adopted the same incomplete list and reported 41/41 runs completed, and the agent trusted that result until the end. In another task, GPT-5.6 Sol noticed its method missed targets but wrote the paper anyway instead of starting over.

Longer runtimes did not lead to better results. DeepSeek V4 Pro and GLM-5.3 ran the longest at median times of 19.5 and 13.9 hours but scored poorly, while GPT-6 Astra had a median runtime of just 1.3 hours and delivered the best results. Because the strongest model also served as evaluator, the researchers ran a cross-check in which the original papers' authors judged anonymized AI papers themselves. On whether a paper matched the given research direction, the automated system agreed with the experts in every case. On scientific quality, judgments aligned in only about two thirds of cases.

Sources: The Decoder