Korva is a scroll of the internet’s best ideas.
GPUs actually suck for deep learning
The pitch
Five founders showed up with a deck, and the venture capitalist who heard it did not want to go to the pitch. This was before transformers. OpenAI was a strange research lab, Nvidia was worth about 40 billion dollars instead of 4 trillion, and the TPU had not been announced. After a first meeting on Wednesday and a partner meeting on Monday, the firm invested.
The three levers
The founders laid out three levers that still speed up deep learning in hardware: more cores, more communication between cores, and memory closer to the compute. Pushing all three to the limit meant a wafer-scale chip with 450,000 cores and about 20 gigabytes of SRAM on the chip, so memory never required leaving it. With every core on the same wafer, communication between them was as good as it could get.
Why hardware is slow
The block diagram was the easy part. In software, a diagram of why it works puts a company about 80% of the way there. In hardware it puts it about 2% of the way, because physics and a supply chain of 30 vendors beyond TSMC still stand in between. The first parts came back around 2019 and went through 14 steps of bring-up. In a 2019 board meeting, after the company had raised about $500 million, the thing was melting, and the investor's first thought was that all that money was gone. By 2020 the first version that worked was running.
Why does moving data cost more than computing?
Mark Horowitz of Stanford showed in a 2014 conference talk that fetching a number from off-chip memory costs far more energy than doing arithmetic on it. A 32-bit addition takes a tiny fraction of the energy of a trip to DRAM. Wires carry signal over distance, and that distance is what burns power and time. Keeping memory on the chip next to the cores cuts out the longest trips.
Why is SRAM faster than other memory?
SRAM stores each bit in a small circuit of about six transistors that holds its value as long as power is on. DRAM stores each bit as charge in a capacitor, which leaks and must be refreshed constantly. That refresh and the extra reading steps make DRAM slower. SRAM is faster but takes more chip area per bit, which is why chips carry far less of it.
Why are wafer-scale chips so hard to make?
Every silicon wafer carries tiny defects, so chipmakers normally cut the wafer into many small chips and discard the flawed ones. A chip the size of a whole wafer has to work around defects instead, by routing signals around bad cores. Gene Amdahl's company Trilogy Systems tried wafer-scale integration in the 1980s and gave up on it. Power delivery and heat across so large an area add further problems.
More from Invest Like the Best with Patrick O'Shaughnessy
