NVIDIA's AI Team Won Just a Sliver of a Chip
Engineers battled the graphics team for 5 to 10 percent of chip space to speed up early AI work, inside a frugal culture obsessed with physical limits.
Invest Like the Best with Patrick O'Shaughnessy · Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]The fight was over real estate you need a microscope to see. Around 2015, inside NVIDIA, a small group of engineers wanted to carve out a piece of the company's gaming chip and give it to artificial intelligence.
Neil Movva, founder of the AI infrastructure startup Sale Research and a former NVIDIA engineer, remembers reading the lab notebooks of his managers from that period. They had been visiting small machine learning conferences and taking notes on a strange trend. Graduate students were buying gaming graphics processors, built to paint pixels on a screen, and using them to train large learning models.
By 2015 and 2016, that observation turned into a bet. NVIDIA started secret side projects to adapt the graphics processor for machine learning, and Jensen Huang, NVIDIA's chief executive, decided to double down on the idea that this use would only grow. The plan was to put the first version of what are called TensorCores onto the gaming chip, specialized circuits that speed up matrix multiplication, the row-by-column math that AI uses to find patterns in data.
That is where the fight started. On any chip, the working area on the slice of silicon, known as die area, is precious, and designers guard it carefully because anything given to one function cannot go to another. The AI group was competing directly against the graphics teams that had made NVIDIA dominant.
"We fought tooth and nail and got just a tiny bit ... maybe like 5%, 10%, something like that," Movva recalls. It was enough to add some acceleration for a basic step used to find shapes in images, the core math for the image-recognition models of the day.
The software team where Movva worked then had to squeeze every bit of performance out of that sliver. That taught him what he calls NVIDIA's ethos of the speed of light, the idea of always pushing hardware to the physical edge of what it can do. If engineers thought a chip could run at a certain frequency and produce a certain number of multiplications per cycle, they broke every bottleneck to get there. Movva says he still tells his own engineers to chase 100% speed of light, and to care about absolute numbers, not how they compare with competitors.
By his account, it was a place with remarkable retention and extremely motivated engineers. Many of the people he worked with in 2015 and 2016 are still at NVIDIA today, engineers he describes as believers in parallel computing who treated the evolving chip as their life's work.
It was also famously frugal. After cutbacks that followed 2008, there were no perks like free lunch. Movva remembers clearly that there was not even free milk in the fridge. Anyone who wanted milk for coffee had to chip in a dollar a month to the milk club, which stocked the fridge with Costco milk. He is quick to add that his own company does not do that.
Questions people ask
Why were gaming chips useful for AI in the first place?
Graphics chips were built to do many simple calculations at the same time to draw images quickly. That same ability to do lots of math in parallel turned out to be useful for training neural networks, which also depend on doing many similar calculations at once.
What does chasing the speed of light mean for a chip?
It is shorthand for the physical limit of what the hardware can do, such as how many calculations it can complete per second at a given speed. At NVIDIA, Movva says, engineers treated that limit as the target and kept removing bottlenecks until they reached it.