Korva is a scroll of the internet’s best ideas.
Memory makers swapped burgers for steaks
Picture a steakhouse with a couple of burners. Samsung, SK Hynix, and Micron can build high bandwidth memory, the kind AI needs a lot of, while players like SanDisk supply regular memory. With limited capacity, switching from low margin burgers to $200 dry-aged ribeyes is the obvious move.
Demand from AI data centers is so large that the makers are focusing on building that out, with less capacity left for the memory in everyday devices.
Prices for those devices are climbing. The budget Neo laptop, marketed as a device that used less RAM, is going from $599 to $699, and a MacBook Pro is going from about $1,700 to about $2,000. iPads, HomePods, and Apple TVs are expected to rise too.
Why is high bandwidth memory harder to make?
High bandwidth memory stacks several memory chips vertically and links them with thousands of tiny vertical connections called through-silicon vias. Each stack then sits right beside the processor on a shared base so data travels a very short distance. The extra stacking and packaging steps lower manufacturing yield compared with a plain memory chip. A single unit also consumes more of the factory's wafer supply than ordinary memory does.
Why can't makers just build more factories quickly?
A modern memory fab costs billions of dollars and typically takes two or more years to build and equip. Chipmaking tools come from a small number of suppliers, and orders for them have long waiting lists. Memory makers have also been through sharp boom and bust cycles, which makes them cautious about adding capacity. The industry is dominated by just three companies: Samsung, SK Hynix, and Micron.
Why do AI data centers need so much memory?
A large language model has to hold billions of numbers, called weights, in fast memory while it generates each word. Moving that data to the processor is often the slowest part of the job, so memory bandwidth limits how fast the model runs. A single AI accelerator card can carry several stacks of high bandwidth memory. Training clusters link thousands of those cards together.
More from Lightspeed Venture Partners
