Korva is a scroll of the internet’s best ideas.
The 2016 pitch
The company was Cerebras, and the pitch landed in 2016, before the Transformer, when OpenAI was a strange research lab, NVIDIA was worth about 40 billion dollars rather than 4 trillion, and the TPU hadn't been announced. GPUs were about 100 times better than CPUs for deep learning, and the founders' point was that they still weren't good.
Three levers
Speeding up deep learning in hardware comes down to three levers: more cores, more communication between cores, and memory closer to the compute. The plan pushed all three to the maximum. A wafer-scale chip held 450,000 cores and roughly 20 gigabytes of SRAM on the chip itself, so data never has to leave it to reach memory, and because every core sits on the same wafer, communication between them is as good as it gets.
First silicon
The first parts came back around 2019, followed by a bring-up process of about 14 steps, and a first system that worked by 2020.
Why does keeping memory on the chip help?
Moving data costs far more time and energy than computing on it. In a widely cited 2014 analysis by Mark Horowitz, fetching a number from off-chip memory used hundreds of times or more the energy of adding two numbers. Memory that sits on the same silicon as the compute avoids that trip. SRAM, the kind used on-chip, is faster than the DRAM that sits in separate packages.
Why aren't chips normally the size of a wafer?
Wafers always carry some defects, and a normal chip is cut from the wafer so that defective ones can be thrown away. A chip the size of the whole wafer is almost certain to contain flaws, so the design has to route around bad cores. Chipmaking machines also expose only a limited rectangle at a time, so connecting wires across the full wafer takes special work. Standard wafers are 300 millimeters across.
What did the Transformer change in 2017?
Google researchers introduced the Transformer in the 2017 paper Attention Is All You Need. Earlier language models processed text one word at a time, which kept them from using chips' parallelism well. The Transformer looks at all words in a sequence at once, so training spreads across many cores. GPT, BERT, and later chatbots are built on it.
More from Invest Like the Best with Patrick O'Shaughnessy
