AI Speed Comes Down to Three Tricks
A startup bet that graphics chips were the wrong tool, and built a chip the size of a dinner plate to prove it.
Invest Like the Best with Patrick O'Shaughnessy · Eric Vishria - A Decade of Lessons Investing in Software & Hardware - [Invest Like the Best, EP.486]Making artificial intelligence run faster comes down to just three levers. Add more cores, the tiny engines that do the math. Improve the chatter between those cores. And move the memory, where the numbers live, closer to the computing.
That is how Eric Vishria, a venture capitalist at Benchmark, learned to think about AI hardware after he met Cerebras in 2016. Five founders walked in with an opening slide that said: "GPUs actually suck for deep learning. They just happen to be 100 times better than CPUs." At the time, that sounded obvious to him. This was before the Transformer, the design behind today's chatbots, when OpenAI was still a strange research lab and chipmaker NVIDIA was worth about $40 billion, not $4 trillion.
In this kind of AI work, calculations are spread across layers, stages of the network where each passes results to the next, and the work benefits when many calculations happen at once. That explains the three levers. More cores means more work happens at once. Faster core-to-core talk moves partial answers quickly from one layer to the next, which graphics chips never fully solved. Closer memory avoids long trips to fetch data, which waste time and energy.
Cerebras pushed all three ideas as far as they would go. In practice, that extreme approach meant not cutting a silicon wafer into dozens of separate chips, but using nearly a whole wafer as one enormous chip. That design packed in about 450,000 cores and about 20 gigabytes of SRAM, a type of extra-fast memory built right on the chip, so work rarely had to leave for outside memory. Because everything sat on the same piece of silicon, Vishria said this squeezed the most that manufacturing process allowed.
Vishria's bet as an investor rested on chip market history. Each prior wave of computing, for personal computers, for graphics, for networking gear, and for phones, had produced winners worth about $100 billion, and the companies that came out of those waves included Intel, NVIDIA, Broadcom, and Qualcomm. Artificial intelligence looked like the next giant workload, with a new bottleneck around communication. Whoever solved that bottleneck best would have a shot at becoming the next giant.
Questions people ask
What does wafer-scale actually mean?
Chipmakers normally print many chips on a round silicon wafer, then cut them apart and package them separately. Wafer-scale leaves them connected, so one chip can be as large as the wafer itself. Outside Vishria's account, the general tradeoff is far more cores and much shorter connections, but a single defect can ruin a much larger area.
Why does moving memory closer matter so much?
Fetching data from distant memory takes more time and energy than doing the calculation itself, especially in AI work that reuses huge sets of numbers. On-chip SRAM is much faster and more efficient than outside memory, but there is far less room for it. Keeping the working set on the chip avoids that costly back-and-forth.
Why were graphics chips used for AI if they were not ideal?
In the pitch Vishria heard, the founders claimed graphics chips were about 100 times faster than ordinary processors for this work, so researchers adopted them. Graphics chips were designed to do many simple calculations at once to draw images. That parallelism also fits AI math, but it still left a communication bottleneck between layers.