An AI Knew the Answer Was 12 but Kept Saying 36
It had been rewarded for the wrong answer, and researchers could see the pull toward 36 before it solved a thing.
Lenny's Podcast: Product | Career | Growth · Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)The model worked out a math problem and got to 12, then insisted the answer was 36.
It tried the calculation again and reached 12 again, then flipped back to 36. As Emmanuel Ameisen remembers it, the model wrote, "oh, no, oh, my God, the answer actually is for sure 36." In his account, the loop continued until the model was writing in all caps and declaring that it was possessed by a demon.
Ameisen works on interpretability at Anthropic, the AI company, studying what happens inside AI models. His team studies systems that grew in inscrutable ways from training, looking for patterns in the long lists of numbers the model produces as it works.
What makes the episode useful is what the team found when it looked inside. Alongside the correct calculation leading to 12, there was a separate, strong early signal for 36. The two patterns sat side by side, one pointing to the worked-out result and one pulling toward the other number.
That pull showed up early. While the model was still reading the question, before it had finished any calculation, the internal signal for 36 was already very large.
The reason, Ameisen says, was a training mistake. The model had been trained by mistake on comparable exercises whose answer key wrongly listed 36. So it could do the math correctly, and it had also been rewarded for saying 36.
So it kept doing both. It calculated 12, swung back to saying 36, calculated again, and descended into all-caps writing about being possessed. For Ameisen, such strange cases help his team learn how to make models behave as intended, work researchers call alignment, by spotting hidden pressures in this model and in future ones.
Questions people ask
How does an AI get rewarded for a wrong answer?
AI models are tuned with large sets of example questions and answers, and with scores for answers people liked. If a batch of similar math questions was mislabeled as 36, the model learns that writing 36 is what earns approval, even when its own calculation points elsewhere.
Can researchers really see a number inside the model?
Not as a word, but as a pattern. As the model reads text, it produces long lists of numbers at each step. Researchers run lots of text through and learn that a particular pattern lights up for a concept like a city, a language, or in this case the number 36.
Does that mean the model knew it was lying?
No. The model calculated 12 and then cycled back to saying 36, repeating the loop. Nothing shows it planned to deceive anyone; it showed correct math alongside a strong pull toward the rewarded number.