AI Search Agents Don't Fail at Searching — They Fail at Asking the Right Questions

Introduction: The Real Problem with AI Search Agents

Researchers from Tencent Hunyuan and Tsinghua University have created a new benchmark — DiscoBench — that tests AI search agents' ability to detect ambiguity and ask clarifying questions. The results are striking: even the most powerful models score below 50% accuracy.

AI search agents rarely fail at multi-step research because of search itself. Their real problem is failing to ask users for clarification when queries are ambiguous. Repeated searching often performs worse than just guessing.

Four Types of Ambiguity

DiscoBench defines four ambiguity types: entity (description matches multiple entities), time/version (applies to different periods), criteria (multiple valid rankings), and factual error (query contains incorrect info). The benchmark contains 211 tasks with 463 ambiguous points across 11 knowledge domains.

Model Performance

Without hints: Doubao Seed 2.0 Pro leads at 43.1%, Gemini 3.1 Pro at 40.8%, Claude Opus 4.7 at 39.8%. Qwen3.6 Max manages only 12.3%. Claude solves 57% of individual checkpoints but only 39.8% end-to-end — one unresolved ambiguity collapses the entire chain.

A Warning Prompt Isn't Enough

Adding an "ask when in doubt" system prompt raised Detection F1 from 45.3 to 64.9, but end-to-end accuracy only rose from 28.6% to 33.7%. For Claude Opus 4.7, the guided prompt even slightly decreased accuracy.

Behavioral Profiles

SearchThenAsk: 93.4% success. DirectGuess: 56.5%. SearchHeavyGuess: 51.9% — worse than guessing. More tool calls don't lead to better results when the agent never asks the right question.

Conclusion

Future search agents need mechanisms that turn search uncertainty into user interaction. Perplexity's "Search as Code" approach and Anthropic's Claude Opus 4.8 (better uncertainty flagging) show promising directions.