Korva is a scroll of the internet’s best ideas.
Split inference across three chips
Three jobs, three chips
People usually talk about prefill and decode, and decode has two parts: attention and the feed-forward network. The holy grail is to run prefill on one chip that has no HBM DRAM, attention on a super high-powered chip with HBM DRAM, and the feed-forward network on an SRAM chip.
Why SRAM chips fit
SRAM-based accelerators are not constrained by HBM DRAM, and many are made on older chip nodes, so they don't compete with the latest GPUs for manufacturing capacity. They can be added to the existing installed base of compute as well as to new builds.
Nothing beats SRAM for the feed-forward network. Workloads keep changing, so no single chip can get the ratio of compute to HBM DRAM to SRAM right, but being able to split the work into three parts should lift the return on AI infrastructure.
Why is SRAM faster than DRAM?
SRAM stores each bit in a small circuit of about six transistors that holds its state as long as power is on. DRAM stores each bit as charge in a tiny capacitor, which leaks and must be refreshed constantly. SRAM can be read quickly and sits right on the processor die, which is why CPU caches are built from it. It takes much more chip area per bit, which is why there is far less of it than DRAM.
Why does attention strain memory?
During generation, a model keeps a record of every earlier token's keys and values, called the key-value cache. Each new token must read that entire cache to decide what to pay attention to. The cache grows in step with the length of the conversation. For long prompts, moving that data takes longer than the arithmetic does.
Why is prefill heavy on compute?
Prefill processes every token of the prompt at the same time, so the chip stays busy with arithmetic. Decode produces one token at a time, and each step must read the model's weights and cache from memory again. The two phases therefore stress different parts of the hardware. Prefill is limited by compute, while decode is limited by memory bandwidth.
More from Invest Like the Best with Patrick O'Shaughnessy
