Korva is a scroll of the internet’s best ideas.
Benchmarks mean nothing on the street
Telling someone on the street that a new model hit a certain score on a benchmark doesn't mean anything to them. Most people don't even know how hallucinations work, so it is hard to make them care about a technology that doesn't touch their daily life.
The unlocks that get the most attention are in deeply technical spaces like coding. Google's storytelling around AlphaFold, and around AlphaGo and AlphaZero, carries further because anyone can see the step change in a game and understand what it means.
The same applies to examples like personalized cancer vaccines and Disney animatronics. They widen the picture of AI beyond a better search engine or an AI girlfriend or boyfriend, which is what AI represents in a lot of people's heads right now.
Why did one AlphaGo move stun Go players?
In March 2016, DeepMind's AlphaGo played Lee Sedol, one of the top Go players in the world. In game two, AlphaGo played its 37th move on the fifth line from the edge, a placement that commentators first took for a mistake. DeepMind later said AlphaGo's own estimate was that a human would play that move about one time in 10,000. AlphaGo won the match four games to one, and Lee Sedol won game four.
How did AlphaFold change biology?
Proteins fold into three-dimensional shapes that determine what they do, and working out a single shape through lab methods could take years. DeepMind's AlphaFold 2 won the CASP14 protein-structure competition in 2020 with accuracy close to experimental methods. In 2022, DeepMind and the European Bioinformatics Institute released predicted structures for more than 200 million proteins. Demis Hassabis and John Jumper shared the 2024 Nobel Prize in Chemistry for this work, along with David Baker.
Why do benchmark scores stop meaning much?
Once top models score near the maximum on a test, the test can no longer separate them, a problem called saturation. Another issue is contamination, where a test's questions leak into training data and inflate scores. The MMLU benchmark, introduced in 2020, was topped by leading models at above 90 percent by 2024. Researchers responded with harder successors such as MMLU-Pro.
More from Sourcery with Molly O'Shea
