How video game chips learned to do AI
After spotting grad students training AI on game cards, NVIDIA gave a tiny slice of its chip to machine learning, then pushed it to its physical limit.
Invest Like the Best with Patrick O'Shaughnessy · Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]In the mid-2010s, managers from NVIDIA began turning up at small machine learning meetings like ICML and NeurIPS. They carried lab notebooks and jotted down what researchers were presenting.
What stood out, recalled Neil Movva, who now builds systems to serve AI models at low cost and was then a young engineer at NVIDIA, was that graduate students were training AI models on cards sold for playing video games. The managers noted that this deep learning thing seemed to be catching on, and decided it deserved a closer look.
This is the story of how a chip made to render game graphics was remade for AI work. The first step was small, setting aside only about 5% to 10% of the chip for the new job.
By 2015 and 2016, Jensen Huang, the chief executive of NVIDIA, had the conviction to double down on that use and put the first TensorCores on the chip. A TensorCore is a dedicated section for doing the grid multiplication math that sits at the center of AI models.
The addition required a fight over space on the chip. Chip designers protect that area closely, because space given to one use cannot be used for another, and the graphics groups were competing to keep it. Movva said they "fought tooth and nail" to win that tiny share for speeding up the image recognition calculations important then.
Once the hardware was set, the task for Movva's software group was to push it as far as the design allowed. Engineers called the goal the "speed of light," the highest output the hardware could in theory deliver, and treated reaching it as a core belief. Movva says he still tells his own engineers to chase that absolute limit, not just beat rivals.
NVIDIA at the time was frugal and its engineers tended to stay. Many people Movva worked with in 2015 and 2016 are still there, engineers he describes as deeply committed to parallel computing. Frugality went so far that there was no free milk for coffee in the office. Anyone who wanted it paid a dollar a month to a milk club, which kept milk from Costco in the fridge.
Questions people ask
Why was a gaming chip any good at AI in the first place?
A graphics chip draws images by doing the same simple math across thousands of pixels at once. That parallel design turned out to be ideal for AI, which also requires doing millions of similar calculations simultaneously to adjust the connections in a neural network.
What exactly does a TensorCore do?
It is a small, dedicated circuit inside newer NVIDIA chips that performs many multiply and add operations together in one step. Since AI training and inference are dominated by that kind of math, doing it in hardware is far faster and more efficient than doing it with general purpose circuits.
Do other chipmakers now do the same thing?
Yes. Every major AI accelerator now includes some version of dedicated matrix math units, including chips from AMD, Google, and startups. The specific designs differ, but the bet that AI needed its own slice of silicon became industry standard.