Is AI Reasoning Right for the Wrong Reasons?

14 insights saved from this article by @technology

@technology· What is AI?
State-of-the-art reasoning models often work because surrounding conventional software guides and verifies outputs in agentic systems and theorem provers.
@technology· What is AI?
In 2025, Kambhampati’s lab replaced correct reasoning traces with incorrect ones without reducing formal reasoning performance. Correct-trace training still produced invalid records.
@technology· What is AI?
Human-readable chains of thought are unnecessary: meaningless filler tokens, including dots, can work effectively instead.
@technology· What is AI?
A model’s hidden reasoning tokens can be arbitrary embeddings, foreign-language fragments, exclamations, or punctuation.
@technology· What is AI?
OpenAI has not publicly revealed raw chains of thought since 2024. Its released traces are human-expert rewrites produced using another OpenAI model.
@technology· What is AI?
A reasoning trace can look like an auditable record of model thought while having no reliable connection to the reasoning that occurred.
@technology· What is AI?
A correct answer followed by plausible thoughts creates the seductive illusion that those thoughts caused the result.
@technology· What is AI?
Calling a program “UNDERSTAND” or “GOAL” can smuggle the missing capability into its name and fool researchers.
@technology· What is AI?
A reasoning model may be simply an LLM with specialized training, not a fundamentally new reasoning mechanism.
@technology· What is AI?
A model can imitate enough reasoning examples to produce a verifiable answer without learning a general reasoning algorithm.
@technology· What is AI?
Large reasoning models may retrieve approximate answers from training data, landing closer to pattern matching than genuine reasoning.
@technology· What is AI?
Most reasoning models probably do not learn algorithms. They exploit larger example sets and increasingly clever reward signals.
@technology· What is AI?
Reasoning models automate chain-of-thought prompting by generating thinking tokens, then feeding those reasoning traces back into the model.
@technology· What is AI?
A black-box AI is useful when outputs are verifiable, but research demands knowing whether it is right for the right reasons.

Post on X

Keep what you found in Is AI Reasoning Right for the Wrong Reasons?

A free account saves any of these cards to your Collection, and Korva resurfaces them so you actually remember.