AI access makes people almost entirely unwilling to say “I don’t know,” study finds

A new preprint published on arXiv finds that merely having access to AI advice nearly wipes out people’s willingness to admit uncertainty. In five experiments with 3,132 participants, volunteers answered six difficult questions and could always decline to respond. The study has not yet been peer reviewed.

The questions concerned fine visual details from films, such as the color of a team’s kit in “Bend It Like Beckham.” The researchers chose them so that the model they used answered almost always incorrectly: such details rarely appear in online text, which makes them prime targets for hallucinations. That design let the authors separate AI use from the quality of the advice and observe what happens when the advice is wrong.

“I don’t know” nearly disappeared

In the first experiment, participants working without AI declined to answer on 36% of the questions; those who could consult the model did so only 6% of the time. A separate replication was even sharper: 44% versus 3%. Pooled across conditions, participants working alone answered correctly 27.5% of the time, while those with AI access managed only 9.2%. Their confidence, meanwhile, nearly doubled. Co-author Valerio Capraro told IBM Think that accuracy fell by a factor of three while confidence rose 2.5-fold.

Money restores part of the caution

In three experiments, correct answers were rewarded with money, wrong ones penalized, and declining paid nothing. Under those conditions participants sought and followed AI advice less often, answered more accurately, and in two experiments became significantly more willing to withhold an answer. Even so, people with AI access remained far less likely to say “I don’t know” than those who worked without it. According to the authors, incentives recover only part of the lost caution.

Why it matters

The researchers did not establish why the effect occurs. One hypothesis is that an AI answer lowers the metacognitive threshold, the level of confidence a person needs before committing to a response. The effect also held when advice appeared automatically, without being requested. That matters as search summaries, writing assistants and autocomplete volunteer suggestions unprompted. Capraro argues that doubt is not a failure of knowledge: “It is often where knowledge begins.” The authors caution that their questions were obscure, the financial stakes small and the model deliberately poor, so results may differ when advice is usually accurate or when decisions carry serious consequences.