Korva is a scroll of the internet’s best ideas.
Critics call it regression to the mean
The statistical critique
The argument runs like this. Take two noisy measures, here someone's skill and their estimate of their skill. If the first is extreme, very high or very low, the second will necessarily trend toward the middle, so the incompetent appear to overestimate and the competent appear to underestimate.
The authors' answer
The effect's authors answer that many standard ways of correcting for regression to the mean exist, from epidemiology, economics, and psychology, and that the critique ignores them. Correcting dampens the effect only a little.
They point to other methodological problems in the critiques. One study asked people to rate their intelligence on a 1 to 25 scale after an IQ test, then assumed that a self-rating of 25 meant they thought they had a 160 IQ and a 1 meant 40. Intelligence is also so broad that it means different things to different people.
Judging versus producing
In another study, researchers statistically separated the ability to judge a performance from the ability to produce one and found that high performers were no better at judging than those at the bottom. Before the separation, though, high performers were better at judging, because expertise aids both skills, and that is the mechanism the original 1999 paper describes. In real life there is no separation.
Why do extreme scores drift toward the middle?
A score combines real ability with luck and noise. Someone who lands at an extreme was probably helped or hurt by chance, and chance does not repeat on the next measurement. Francis Galton noticed the pattern in the 1880s, when he found that very tall parents had children who were taller than average but shorter than the parents. He first called it regression towards mediocrity.
How can regression to the mean fool experts?
Daniel Kahneman described teaching Israeli Air Force flight instructors who believed that praise hurt pilots and scolding helped them. A cadet praised after an unusually good landing usually did worse next time, and one scolded after a poor landing usually did better. Both changes are what regression predicts, whatever the instructor said. Kahneman recounted the episode in his 2011 book Thinking, Fast and Slow.
More from Matthew Hutson
