Answering an AI prompt is two jobs, loading up and then writing out
First the system prepares its memory of your prompt, then it writes, and speed depends on how fast chips can borrow memory from each other.
Invest Like the Best with Patrick O'Shaughnessy · The Two Harvard Dropouts Who raised $800M to take on NVIDIAAsk a chatbot a question and it does two jobs in sequence. First it takes in your prompt and gets its memory ready, a step engineers call prefill. Then it uses that memory to write out a reply one token at a time, with tokens meaning small chunks of text, roughly parts of words, in a step called decode.
In prefill, the system reads a large amount of text and prepares what is called a KV cache, essentially the model's short term memory of what matters in that text. In many large systems, that memory is then sent to a different group of chips that handles only decode. One way to put it is loading a gun and then firing it, a comparison raised in the conversation that Gavin Urtz, a co-founder of the chip startup Etched, which is building computers designed only to answer AI queries, agreed with.
Decode is a memory game. To produce each next token, chips must pull from that memory very quickly. The natural question sounds like how fast is one chip's memory, but Urtz says the right question is how fast is the memory of the whole cluster working together. If chips can borrow memory from each other quickly, they can serve more tokens per second for each user.
The delay between chips keeps that from working well today because on Nvidia's Blackwell chips, a hop from one chip to another takes about 4,000 nanoseconds, so eight chips working together deliver far less than eight times the speed of one. Etched's bet, which it calls cluster scale memory, is a custom communications system that cuts that delay and bandwidth bottleneck by more than five times, letting memory spread across chips act more like one pool.
There is still enormous headroom, since the physical limit is the speed of light, which could in principle move data chip to chip in just two or three nanoseconds. Looking 20 or 30 years ahead, Urtz compares the scale to a chip factory, a single $40 billion building with just a few production lines. He envisions a similarly large AI installation, costing $40 billion to $100 billion, devoted to serving one or a few models to huge numbers of people at once to get the same economies of scale.
Questions people ask
Why put prefill and decode on different groups of chips?
Prefill needs huge amounts of math to absorb a prompt at once, while decode needs huge amounts of memory access to drip out one token at a time. Running them on separate groups of chips lets each group be tuned for its own job, rather than forcing one design to compromise.
How could chip to chip messages ever take only a few nanoseconds?
Light travels about a foot in a nanosecond, so even chips sitting inches apart cannot talk instantly once packaging, wires, and switching are included. Two or three nanoseconds is a theoretical floor, and real systems add overhead, which is why moving from 4,000 toward tens or hundreds would already be a big win.
What is Etched actually building, just a chip?
No, it describes a full inference system, including the chip, the board it sits on, power delivery, the connections between chips, and the rack that holds them. The argument is that production at large volume is the product, because the advantage comes from the whole system working together.