An AI Assistant Can Make You More Confident Even When You Are Wrong
In one study, accuracy rose 25 points when the AI was right and fell 15 points when it was wrong, and users followed it both times.
mondaymorning.substack.com · The Attention ProblemIn a study called “Cognitive Surrender,” people given an AI assistant went along with its answers to the questions they were asked, whether the AI was right or wrong.
When the AI was right, the participants’ accuracy on those questions rose by 25 percentage points. When the AI was wrong, accuracy fell by 15 points. They checked the machine’s work very little, setting aside both first instincts and careful thought.
Even when they ended up with wrong answers the AI had supplied, participants said they felt more confident. Handing over the thinking felt good.
The takeaway is plain: a confident feeling after using AI does not mean the answer is correct.
Questions people ask
Who ran the Cognitive Surrender study?
Shaw and Nave at the Wharton School ran it. They gave people access to an AI assistant and measured accuracy and confidence on the questions in the study.
Does this only happen with AI chatbots?
It is not limited to chatbots. Research on automation bias has found that people often follow GPS directions, cockpit alerts, and other automated prompts even when other evidence points the other way.