Korva is a scroll of the internet’s best ideas.
Mythos trained on under 200 megawatts
Where compute really goes
A lab's compute splits roughly 60% training and 40% inference, and training splits again: about 50% of the total goes to research and 10% to development. Research means testing new architectures, data mixes, hyperparameters, and attention techniques. The final run takes only a sliver. The reinforcement learning that followed used even less at a single site, and the stages run in sequence, so at most about 200 megawatts was ever working at one moment.
Why runs stay small
Reasons for the cap exist. Clusters are hard to coordinate and co-locate, multi-site training is hard, and generating more rollouts during RL does not necessarily make a model better.
The gap with China
That is why compute gaps with China matter little for now. The leading Chinese labs have 100 to 200 megawatts total, ByteDance's Seed is the outlier with significantly more, and Kimi runs nowhere near a gigawatt, while Anthropic reaches nearly five gigawatts by the end of the year. As automated coding, automated researchers, and continual learning arrive, the line between research and training blurs, and more of the compute budget flows into training the model.
Why is training across several sites so hard?
In large training runs every chip must share its results with the others after each step, which takes huge bandwidth and low delay. Light in fiber covers about a kilometer every five microseconds, so even short distances add waiting that thousands of idle chips make expensive. Clusters spread across sites also have to survive link failures mid-run.
How does reinforcement learning use compute differently?
Pre-training is one synchronized job over a fixed dataset. Reinforcement learning has the model attempt tasks, scores the attempts, and uses the scores to update the model. The attempts can be generated on many separate machines, but the updates still depend on them, and more attempts do not always yield a better model.
Why do labs run so many small experiments?
A full-size training run is too costly to repeat, so researchers train many small models first. They fit scaling curves to the results and extrapolate how a much larger model should behave. Most of the compute in research goes to these tests of architectures, data mixes, and settings, not to the final run.
More from Dwarkesh Podcast
