Korva is a scroll of the internet’s best ideas.
The bottleneck keeps moving
How the bottleneck moved
Seven or eight years ago, all the compute in the world would not have helped, because the missing piece was the research idea. Then the path was clear and only compute held things back, so the work was scaling up. After that the data ran out, and a new approach had to be found for that.
Today compute is still the bottleneck, and roughly the last six months have been a real triumph for research ideas again.
Why compute and ideas link
The two are less separate than they sound, because more compute lets researchers try more things. One recent statistic: the biggest de-risking runs for upcoming training runs are now as big as the entire compute run from 18 months ago, or about that. There is always a bottleneck, and it moves around.
Why do scaling laws make compute matter?
In 2020, OpenAI researchers led by Jared Kaplan reported that language model loss fell as a smooth power law as compute, data, and model size increased. The trend held across more than seven orders of magnitude of compute. That predictability let labs forecast what a larger run would achieve before paying for it. DeepMind's 2022 Chinchilla paper later showed that models and data should grow together, so a bigger model with too little data wastes compute.
Why can AI training run out of data?
Large language models learn from text, and the supply of quality public human-written text is finite. Epoch AI, a research group that tracks AI trends, projected in 2024 that models could use up the stock of public text between 2026 and 2032. Labs have responded in part by generating synthetic data and by training with reinforcement learning on tasks where answers can be checked. DeepMind's AlphaGo Zero showed the underlying idea in 2017, learning Go from self-play with no human game records.
How does one idea change a whole field?
In 2017, Google researchers published Attention Is All You Need, which introduced the transformer architecture. Earlier language models read text word by word, while transformers process all positions in parallel, which made training on large clusters of chips practical. Nearly every major language model since has been built on that design.
More from Invest Like the Best with Patrick O'Shaughnessy
