Korva is a scroll of the internet’s best ideas.
Brains beat data centers on energy
Using the result costs energy too. Each time it answers a question, it needs about the power of a thousand American homes to operate at the level of a human mind.
Those figures are improving fast. Inference is getting five to 10 times more efficient every 12 to 18 months, driven by chips from Cerebras, Groq, and Nvidia and by better models.
Energy is a limit on how fast AI can grow, and the human brain runs on a sliver of that power. The most computationally efficient intelligences on planet Earth are still human ones, so there will be a use for them.
Why does a brain use so little power?
The human brain runs on roughly 20 watts, about the draw of a dim light bulb, while using around a fifth of the body's resting energy. Neurons fire sparsely, and most signaling happens only when something changes. Memory and processing sit in the same place, the synapse, so data is not shuttled back and forth. A conventional chip spends much of its energy moving data between memory and compute.
How does inference get cheaper so quickly?
Quantization stores a model's numbers in fewer bits, such as 8 or 4 instead of 16, which cuts memory use and energy per answer. Distillation, introduced by Geoffrey Hinton and colleagues in a 2015 paper, trains a small model to imitate a large one. Speculative decoding lets a small model draft text that a large model checks in a single pass. Researchers at Google described speculative decoding in 2022.
How do Cerebras and Groq chips save energy?
Cerebras builds a processor the size of an entire silicon wafer, and Groq builds chips with large pools of on-chip memory. Both aim to avoid trips to external memory. Mark Horowitz of Stanford showed in a 2014 conference talk that fetching data from external DRAM costs far more energy than doing arithmetic on it.
More from All-In with Chamath, Jason, Sacks & Friedberg
