KorvaJoin the waitlist
Tech

Words Like Reasoning Make AI Sound Human, Even to Researchers

A 1976 warning about wishful names helps explain why AI vocabulary invites belief before proof.

AI image
Quanta Magazine · Is AI Reasoning Right for the Wrong Reasons?

In the summer of 2024, shortly before the first of the newer reasoning systems appeared, Melanie Mitchell, an artificial intelligence researcher at the Santa Fe Institute, shared an old phrase with a science journalist who has covered AI for a decade and writes a column about his reporting: wishful mnemonics.

The label dates to a 1976 paper by Drew McDermott titled Artificial Intelligence Meets Natural Stupidity. McDermott warned that names such as UNDERSTAND or GOAL can make simple software sound thoughtful.

McDermott wrote, "If a researcher ... calls the main loop of his program UNDERSTAND, he is (until proven innocent) merely begging the question."

He added, "He may mislead a lot of people, most prominently himself." His safer test, he said, was to call the loop G0034 and then try to show what understanding G0034 actually implements.

The columnist says that warning fits current systems. Mitchell told him these models get more answers right when they produce intermediate text before answering, but the text is not necessarily faithful to what happens inside the model, and much of it can be removed without hurting results.

Mitchell said, "We react to language in a way that is very anthropomorphizing. That's just the way that we humans work."

That same AI columnist applied McDermott's point about words that make machines seem to think to today's AI vocabulary of reasoning, chains of thought, and thinking tokens. By chains of thought and thinking tokens, he means the written-out reasoning text a model produces before its final answer. He called that vocabulary "wishful mnemonics all the way down," describing it as "a heady mix of shorthand and suspended disbelief." He added that new areas of research may need some version of that mindset to get started.

He wrote, "I don't foresee always reaching for the air quotes around AI reasoning, any more than I'd put them around the horse in horsepower." He compared the models to engines that need fuel, give off exhaust, and run fast, and added of his Toyota that "it's not because I believe there are little hooves pounding away under the hood."

Questions people ask

If the thinking text isn't faithful, why do AI companies show it to users?

Partly for transparency and partly for usefulness. The text can help users follow an answer and spot mistakes, even if it does not record the model's internal calculations. Companies now often show a rewritten summary rather than the raw internal text.

Does this mean AI doesn't really reason?

No one has settled that. Critics say the models match patterns in a way that looks like reasoning, while defenders say solving novel math problems should count as reasoning. The dispute is about what the word reasoning should require.

What is a large reasoning model, in plain terms?

A large reasoning model is a large language model trained to write extra intermediate text, also called reasoning traces or thinking tokens, before its final answer. That text is fed back into the model to guide it, with extra training that rewards answers that prove correct.

More in Tech