AI's Written Explanations May Not Be Why It Gets Answers Right
Swapping an AI's written steps for dots or errors still yields right answers, suggesting the text may not drive the result.
Quanta Magazine · Is AI Reasoning Right for the Wrong Reasons?AI's written explanations may not be why it gets answers right. Replace that step-by-step text with dots or deliberate mistakes, and models still land on the right answer just as often.
Researchers at New York University found that rows of dots with no meaning could stand in for a plain-English step-by-step write-up. William Merrill, a professor at the Toyota Technological Institute at Chicago and a coauthor of that 2024 study, put it plainly: "There's no guarantee the chain of thought has to be meaningful in any sense."
Subbarao Kambhampati, an artificial intelligence researcher at Arizona State University, found the same pattern in 2025 when his lab swapped correct reasoning traces, the written-out steps the model produces, for incorrect or irrelevant ones without hurting performance.
A 2025 study of open models from Northeastern University and the University of California, Berkeley found that 30 to 60 percent of thinking steps had minimal impact on math answers. Cutting about half of those steps barely hurt performance.
Questions people ask
If the written steps don't matter, why does asking AI to think step by step improve answers?
Researchers say the extra text gives the model more room to work before it answers. That extra work can raise accuracy even when the words themselves are not a faithful record of how the answer was produced.
Does this mean AI models aren't really reasoning?
Not necessarily. In general, these results mean the written steps may not reflect what drove the answer. So reading the explanation alone cannot settle whether the model reasoned.
Can I trust the explanation an AI chatbot shows me?
Treat it as a helpful draft, not an audit trail. It often sounds plausible even when it does not match how the answer was produced, so check important math, code, or facts with an outside source or a tool that can verify them.