Korva is a scroll of the internet’s best ideas.
Crazy results fail replication most
A failed replication
A recent attempt to replicate an experiment claimed that people learn more quickly when an EEG machine synchronizes their brain waves to an external stimulus. The replication, run by Sasha Putilin on a grant meant to double-check the original, found no effect. The original had been hyped by the media with the promise that you can learn things twice as fast just by changing your screen flicker.
Why surprise predicts failure
Results like that spread because they contradict everything normal people believe, and the same quality is what makes them likely to fail. Ordinary progress in a field is a small step on top of work that has already held up. A claim that flips common sense has no such foundation under it.
Where else it happens
Social psychology fell into this trap especially hard because it naturally touches on topics of popular interest and political explosiveness. A claim like people only perform badly because society tells them they will perform badly spreads fast for the same reason. No subfield of psychology is entirely immune, and economics, medicine, sociology, political science, and some of the hard sciences each hit their own replication crisis at a different time.
Why do surprising results replicate less often?
A claim that contradicts well-tested findings starts with a low prior probability, so one study showing it is weak evidence that it is true. The Open Science Collaboration's 2015 project repeating 100 psychology studies found that original effects rated as more surprising were less likely to replicate. Findings that extend existing work sit on results already confirmed many times. A single dramatic result has nothing comparable behind it.
How does publication bias inflate striking findings?
Studies that find an effect are more likely to be written up and accepted by journals than studies that find nothing. In 1979, the psychologist Robert Rosenthal called the unpublished null results the file drawer problem. When the null results stay hidden, the published literature shows only the successes. A reader of that literature sees a pile of positive findings and has no view of the failures that would have balanced them.
Did medicine have its own replication crisis?
In 2011, researchers at Bayer HealthCare reported that they could reproduce the published findings in only about a quarter of the preclinical projects they examined. In 2012, Glenn Begley and Lee Ellis of Amgen reported that they confirmed the main findings of only 6 of 53 landmark cancer studies. Both reports involved early laboratory research that drug companies use to decide which ideas to develop. Results of this kind pushed journals and funders in biomedicine to revisit how they check and share data.
More from Scott Alexander
