A Startup Wants to Stack Memory on Top of the Processor
Math power has grown a millionfold, memory speed only 40fold, leaving AI waiting on data.

The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch · 20VC: Taking on NVIDIA: Fractile CEO on the Future of Chips | What Everyone Gets Wrong About Capex Spending | The Winners and Losers in Neo Clouds with Walter GoodwinOver the past 25 years, the math power of AI chips has grown about a millionfold. The speed at which memory can feed them data has grown only about 40fold.
That growing gap now limits how fast chatbots and language models run. Chips can calculate far faster than copper wires can deliver the data they need.
The bottleneck is clearest when a chatbot answers. To generate text word by word, the processor must repeatedly load the model's weights and all the context so far from memory.
That transfer can set the generation speed. A very large frontier model that could produce about 80 tokens per second, where tokens are roughly word pieces, would be limited by memory bandwidth, not by mathematical capacity.
The industry has lived with two imperfect kinds of memory, says Fractile CEO Walter Goodwin. Fractile is a chip startup focused on AI inference, the stage when a trained model generates answers.
One kind, called SRAM, sits on the processor itself and moves data very quickly. But it stores less than one tenth as much per area as the alternative, and therefore costs more per unit of capacity.
The other kind, called DRAM, is dense, cheap, and built in separate factories on separate silicon. It sits beside the processor and sends data inward over a few thousand copper wires crowded along the chip's edge, an area engineers call the beachfront.
That leaves current chips boxed in, Goodwin argues. Chips that use high bandwidth memory beside the processor share essentially the same limits on speed and calculation, while SRAM based chips move data fast but run into severe capacity limits.
Fractile wants to reject that tradeoff by stacking memory directly on top of the logic that does the calculating. Instead of every signal squeezing through the edge, the entire memory layer would sit above the processor, connected by millions of short vertical wires running straight down.
The goal is SRAM like bandwidth with DRAM like capacity and scale. An existing memory stack cannot simply be placed on a processor, Goodwin says, so the company has been developing manufacturing methods for that three dimensional structure.
If it works, and can be improved over several chip generations, one chip could hold large models and feed them quickly at the same time. Goodwin says the company still expects to adapt the design as AI workloads change.
Questions people ask
What is AI inference?
Inference is when a trained model generates an answer, such as writing text or predicting the next word. It is distinct from training, which is the earlier and often one time process of teaching the model from large amounts of data.
What is high bandwidth memory?
High bandwidth memory is a stack of DRAM chips placed beside a processor and connected by a wide, fast interface. It moves data faster than conventional memory, but it still reaches the processor through connections around the chip's edge.
Why not just add more wires around the edge?
Wires can be made smaller, but only up to physical and manufacturing limits, and there is limited space along the edge of a chip. Stacking memory on top would use the whole area of the chip for connections, allowing far more wires than the perimeter can fit.
