Korva is a scroll of the internet’s best ideas.
What distillation is
Sonnet is a partially distilled version of Opus. This is how smaller models are made out of bigger ones, and it is a way to teach a model something new found elsewhere in the ecosystem.
Open models as teachers
Most Chinese open-weight models allow distillation, which means their outputs can feed reinforcement learning on top of a model someone else is building. That is one route for American labs to catch up. Because the outputs are visible, they can also be inspected for conflicts with the voice or constitution of the model being created, and reinforcement learning rollouts give a much better shot at catching them.
What the terms allow
Labs have a right to forbid it in their terms of service and to cut off access to someone building a competitive model. Most frontier labs don't appear to prohibit building a smaller model focused on one specific, non-competitive job. There are markets for companies that allow distillation and for companies that don't.
How does distillation pass knowledge to a smaller model?
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean described the method in a 2015 paper called Distilling the Knowledge in a Neural Network. A small student model is trained to match the full probability scores a large teacher gives to each possible answer. Those scores carry more information than a single right answer, such as which wrong answers are nearly right.
How can one model's outputs train another?
Outputs from a strong model can serve as training examples for a student model, or as candidates scored by a reward signal in reinforcement learning. DeepSeek's R1 paper in early 2025 reported fine-tuning smaller Qwen and Llama models on reasoning samples generated by R1. The students improved sharply on reasoning benchmarks compared with their original versions.
How do terms of service restrict distillation?
Terms of service are contracts between a provider and its users. Many providers include a clause barring use of their outputs to build a competing model. The usual enforcement is suspending or cutting off the offending account.
More from 20VC with Harry Stebbings
