Slow AI Is Cheap AI
Background agents that work overnight let data centers pack AI jobs like a bus, not a taxi, and keep expensive chips full.
Invest Like the Best with Patrick O'Shaughnessy · Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]The cheapest way to run artificial intelligence may be to slow it down.
Neil Movva, founder of Sale Research, a company that sells computing for open source AI models by the token, argues that a graphics chip is happiest when it acts like a workhorse, not a racehorse. A GPU, the chip that runs most AI, is fundamentally a throughput machine. It wants a big pile of work that lets it keep all of its computing units busy at once.
For the past two years, he says, the industry has used it the opposite way. The dominant product was the interactive chatbot, which must keep "actually spitting answers out to the person at the keyboard as quickly as possible," as Movva puts it. That keeps the chip underused, because rushing one answer leaves much of its capacity idle while it waits for the next word to emerge.
The alternative is to run wide and slow instead of narrow and fast. The key tool is batching, which means grouping the work of many users together so the chip processes it in parallel. A large batch does more total work in the same time, even though each individual request takes longer, because it rides along with everyone else.
Movva compares it to getting downtown. A private car takes you straight where you want to go. A shared bus picks up many riders, stops along the way, and takes a slower route for everyone. His company, he says, is trying to build the most efficient version of that shared ride from existing chips.
That bus only makes sense if riders stop minding the slower trip. That is why Movva is betting on long running background agents that work for hours or days, doing research or computer chores overnight. If no one is waiting, 10 tokens per second, or about seven words per second, is fine, and the savings from packing the chip full can make intelligence cheap enough to use all the time.
Questions people ask
What does 10 tokens per second actually feel like?
A token is a chunk of a word, roughly three quarters of an English word, so 10 tokens per second is about seven words per second. That is slower than the fastest chatbots, which can feel instantaneous, but faster than most people read carefully. For an agent working overnight, the speed hardly matters because you read the finished report in the morning.
Why can't a chip be both fast for one person and efficient for many?
It is a classic tradeoff found across computing. To answer one request instantly, the chip must leave capacity empty and ready, which wastes its power. To do the most total work, it must stay full by grouping jobs, which forces each job to wait its turn. You can tune for one goal or the other, but not both at peak.
Will chatbots disappear if slow AI wins?
No, instant answers will still matter when you are sitting and waiting. Movva expects the mix to shift, with most computing moving to background work that no one watches in real time. In that world, providers can reserve a smaller, more expensive fleet for urgent chats and run everything else on the cheaper, slower bus.