Korva is a scroll of the internet’s best ideas.
The Forbes investigation names Tencent, ByteDance, Alibaba, and Moonshot among the Chinese buyers. On the US side, the same data sets go to OpenAI, Anthropic, and federal agencies. The material is PhD-written content, reinforcement learning data, and knowledge pipelines, which Forbes calls secret sauce.
The report puts spending by the top six AI labs in China at $500 million a year on this material.
Micro1, another data company, is not among those selling to China: its founder decided not to participate.
Why do AI labs buy human-written training data?
After a model learns from broad text, labs refine it with examples written by people and feedback from people. OpenAI's 2022 InstructGPT paper described hiring human labelers to write model responses and to rank outputs, then training the model on that feedback. Expert-written material is valuable because experts can catch errors in specialized fields that a general reader would miss. The method is known as reinforcement learning from human feedback.
How does distillation pass abilities between models?
A small student model is trained to imitate the outputs of a larger teacher model. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean described the technique in a 2015 paper. The student learns from the teacher's full pattern of answers, not only from a single right answer, which transfers more information per example. Distillation lets a compact model approach a larger model's performance at a fraction of the running cost.
How do US export controls apply to AI chips?
The rules rest on the idea that computing power determines how capable a model a country can train. In October 2022, the US Commerce Department issued rules restricting sales of advanced AI chips and chip-making equipment to China. It broadened those rules in October 2023.
More from All-In with Chamath, Jason, Sacks & Friedberg
