New Taste-Bench Finds Top AI Agents Barely Beat Coin Flip on Long-Term Decisions

Agents struggle to judge long-term decisions

AI agents now handle multi-step tasks such as writing research papers or building software projects, making constant choices about which hypothesis to test, which code to build on, or which experiment to run next. Those choices rarely show their effects right away. A bad decision usually looks fine at first, and its true cost only shows up after the agent has burned through most of its budget. Researchers from Microsoft and City University of Hong Kong call the ability to make these choices well an agent's "taste."

How Taste-Bench works

Existing benchmarks only measure whether an agent finishes a task, not the quality of its intermediate steps. Human grading would be expensive and hard to scale, so the researchers rely on hindsight. They take recorded traces of agent runs, freeze them at a "decision fork" where two attempts diverge, hide all later work, and ask a model to pick the better path. Because the final result of each run reveals the better option, the runs themselves provide a labeled comparison. Taste-Bench contains 502 such questions built from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's public transcript release MALT. A generator model filtered out trivial or undecidable cases, and human review confirmed almost all auto-generated labels.

Even top models barely beat guessing

The team tested 14 models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Each question was asked twice with the options flipped, and counted only if the model got it right both times, giving pure guessing a 25 percent score. The best model, GPT-5.6 Sol, averaged 59.7 percent, closely followed by GPT-5.5 at 59.5 percent. Accuracy dropped to 21 percent, near random guessing, when the key clue only appears after extensive later work, compared with 62.3 percent when the clue is already visible in prior steps. Increasing the reasoning budget barely changed accuracy, suggesting the needed clue simply is not there yet at those forks.

Taste can be trained

The researchers distilled the judgment of a teacher model into Qwen3.6-27B, raising its accuracy on unseen tasks from 30 to 47.9 percent. Used as advice during a separate agent's runs on 41 held-out SWE-bench Pro tasks, the guidance lifted the success rate from 14.6 to 33.7 percent, close to the 39 percent that perfectly correct advice would have achieved. The benchmark and code are publicly available.

Sources: The Decoder