Korva is a scroll of the internet’s best ideas.
Data as a complement
A complementary good is one whose demand rises with demand for another good. Gas is a complement to cars. Data is a complement to model scaling, and the more models get trained, by labs and by companies building their own, the more data the whole industry has to buy.
Data is also harder to replace than GPUs. For data to become irrelevant, humans would have to become irrelevant, which would mean AGI has arrived. Frontier labs spend about 10 to 20% of what they spend on GPUs on data.
The hardest step
Data plus GPUs makes a model, and the algorithms have become somewhat of a commodity because everyone knows how to use the transformer. Sourcing data is the hardest and dirtiest step, and nobody wants to hire the people who generate it.
The market ahead
Anyone who expects GPU spending and model scaling to keep climbing should expect the data market to reach at least $100 billion by 2030, and possibly a trillion.
Why are algorithms now a commodity?
Google researchers published the transformer architecture in the 2017 paper Attention Is All You Need, and the design was public from the start. Nearly every large language model since then uses a variant of it. Improvements now come mostly from scale, data quality, and training tricks rather than from a secret architecture. The paper has become one of the most cited in the history of machine learning.
How do labs get human-written training data?
Labs pay contractors to write answers, rank model outputs, and grade responses. OpenAI's 2022 InstructGPT paper used a team of about 40 contractors to write demonstrations and rank outputs, and that approach became a template for later models. Specialist data, such as expert-written solutions in medicine or law, is priced above general data. Several vendors now exist whose entire business is supplying it.
Why might public text run out?
Epoch AI, a research group that tracks AI trends, estimated in a 2024 paper that the stock of public human-written text could be fully used for training sometime between 2026 and 2032. Models consume far more text each year than people publish. That gap pushes labs toward licensed data, synthetic data, and newly commissioned human work.
More from 20VC with Harry Stebbings
