KorvaJoin the waitlist
Tech

Chips Learned Math a Million Times Faster, but Still Move Data Like Molasses

Each new word forces the processor to reload the entire model, and there are only a few thousand copper wires to carry it.

AI image
The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch · 20VC: Taking on NVIDIA: Fractile CEO on the Future of Chips | What Everyone Gets Wrong About Capex Spending | The Winners and Losers in Neo Clouds with Walter Goodwin

Modern AI chips can do about a million times more math per second than chips from 25 years ago. But their ability to shuttle data in and out has grown only by a factor of around 40 over that span.

That gap is why a chatbot answers the way it does, one word dribbling out after another. It is not waiting on arithmetic. It is waiting on delivery.

Every chip has two headline talents, says Walter Goodwin, chief executive of chip startup Fractile. One is flops, or floating point operations per second, a measure of raw mathematical power. The other is memory bandwidth, or how quickly data can be carried between the processor doing the math and the memory storing the model.

Math has scaled beautifully. Bandwidth is more limited by physics, Goodwin says, lagging by roughly 25,000-fold in relative terms. So more and more AI work is bottlenecked by moving data, not crunching it.

Nowhere is that clearer than in inference, the stage when a finished model generates text. To produce each token, a rough equivalent of a word, the chip must load all of the model’s weights plus the conversation so far. Then it does it again for the next word, and the next.

That repeated hauling explains why a very large frontier model might generate around 80 tokens per second. The processor could do far more math, but it cannot reload the whole model from memory many more times per second than that.

Zoom in and the problem is almost embarrassingly physical. Data travels over copper wires running from external memory into the processor. Engineers have made the wires smaller, but a chip still has only a few thousand of them.

Chip designers have two kinds of memory to work with. Think of external DRAM as a warehouse across the street, dense and capacious, and the processor as the office where the work happens. Workers must carry boxes down the stairs, do the math, and carry them back because the office cannot store everything.

The alternative is SRAM, or boxes kept permanently in the office. It sits on the same silicon as the processor and is fast to reach. But it is not built for density, offering less than one-tenth as much storage per area as DRAM, and therefore costing much more per unit of capacity.

That trade-off is the trap. More math does not make chatbots much snappier if the weights still have to commute from the warehouse for every word. Speeding up AI means speeding up the commute.

Questions people ask

What is the difference between bandwidth and flops?

Bandwidth is the rate at which a chip can move data between its memory and its processor, usually measured in gigabytes per second. Flops measure how much math the processor can do once the data arrives. When bandwidth is the bottleneck, a faster calculator just waits longer.

Why does a chatbot have to generate one token at a time?

A token is the chunk of text a model reads or writes at a time, often a whole word or part of one. Chatbots generate one token, feed it back in as context, and then generate the next. That is why output appears sequentially.

What are DRAM and SRAM?

DRAM is separate, specialized memory built for density, so it can store huge models cheaply but sits farther from the processor. SRAM is built onto the processor itself, so it is much quicker to access but holds far less for the same area and cost. Designers must choose between capacity and speed.

More in Tech