Korva is a scroll of the internet’s best ideas.
AI training outgrows single data centers
Splitting training
Historically, AI systems had to be trained in one room in one building. Google is believed to have been the first to break that up, training across two buildings in one data center area. Now the talk is of training across regions about 100 kilometers apart, and then 1,000 kilometers apart.
A company such as Microsoft could then put one cluster beside the Diablo Canyon nuclear plant and another beside Three Mile Island and have two gigawatts to work with. OpenAI, Anthropic, Google, and Meta are all betting that scaling laws keep holding, and if they do, training at that continental scale has to come online to reach the next level.
The bottlenecks
Supply chains set the pace. Data centers need the right steel, wiring, and fiber optics, and there is a global fiber optic shortage because buyers are ordering thousands of kilometers of it. Generators are another gap, since only so many are ordered each year and Elon Musk, Meta, and Microsoft each want their own on site.
Energy cannot be spun up at gigawatt scale overnight, which is why gas turbines and a restarted Three Mile Island are in the picture. The lag looks like about 18 months before supply catches up with demand, and then the next bottleneck appears, maybe in TSMC chip output or the machines ASML builds.
Why can't one site supply two gigawatts?
A gigawatt is about the output of one large nuclear reactor. New power plants and data centers connect to the grid through an interconnection queue, and Lawrence Berkeley National Laboratory's Queued Up reports found that projects completed in 2022 had waited a median of about five years from request to operation. Transmission lines and substations add their own years of permitting and construction.
How can distant sites train one model?
Training splits a model across thousands of chips that regularly exchange updates, so the delay between sites matters. Light in optical fiber travels about 200,000 kilometers per second, which puts a trip of 1,000 kilometers at roughly five milliseconds one way. Researchers at Google DeepMind published DiLoCo in 2023, a method where groups of chips train on their own and share updates only occasionally, which cuts the traffic between sites.
Why would Three Mile Island reopen?
Three Mile Island has two units. Unit 2 had the partial meltdown in 1979 and stayed shut, while Unit 1 ran until closing in 2019 for economic reasons. In September 2024, Constellation Energy announced a plan to restart Unit 1 under a 20-year power purchase agreement with Microsoft.
More from The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
