284 Experts Made 82,361 Predictions. Random Guessing Did Better.
Even in their own specialties, the experts were no better than nonspecialists.
Veritasium · The Expert MythFor 20 years, political scientist Philip Tetlock asked 284 professional pundits to predict the future.
The group drew on reporters, economists, diplomats, and intelligence staff, most with postgraduate degrees. He asked them questions like whether George Bush would be reelected, whether apartheid in South Africa would end peacefully, whether Quebec would secede from Canada, and whether the dot-com bubble would burst. Each time, the experts put odds on how things might turn out, building a database of 82,361 predictions.
They did worse than if they had simply assigned equal chances to every outcome. Even on their home turf, the experts did no better than people outside their field.
The reason is repetition. Most of the events were one-offs, things the pundits had never lived through many times before, so experience never turned into expertise.
Questions people ask
What does it mean to do worse than equal probabilities?
Assigning equal probabilities means saying every listed outcome is equally likely, without using any special knowledge. It is a very low bar because it requires no insight at all. Doing worse suggests the experts were not just uncertain, but systematically overconfident or misleadingly precise.
Has anyone found forecasters who do better?
Later, Tetlock ran tournaments to find the best forecasters and identified a small group he called superforecasters. They tended to update their views often, think in probabilities, and break big questions into smaller parts. Their success suggested forecasting can improve with the right habits and feedback, even if typical punditry does not.
Why do chess masters succeed where pundits fail?
Expertise works best where the same kinds of problems repeat and feedback is fast and clear. Chess players see similar positions repeatedly and learn quickly whether a move wins or loses. Political events are often unique, with slow and messy feedback, so patterns are much harder to learn.