The same AI model runs five times faster depending on who runs it
The gap reflects specific expertise in running giant models, not just buying more servers.
Invest Like the Best with Patrick O'Shaughnessy · Eric Vishria - A Decade of Lessons Investing in Software & Hardware - [Invest Like the Best, EP.486]According to Eric Vishria, an investor at the venture firm Benchmark, the same freely available AI model running on the same NVIDIA chips can run five times faster depending on who runs it.
By his account, Fireworks, an infrastructure startup that hosts giant AI models for other companies to use, outperforms big cloud providers by about five times on speed alone when offering the identical open source model on identical hardware.
The models are enormous, with two trillion to four trillion internal settings apiece. Simply running them is difficult, and running them efficiently is still harder. Beyond the visible speed gap sits a hidden gap in throughput, or how much work each machine handles, which shapes the underlying economics.
From the outside, the work can look like simple reselling, where sheer size wins. His takeaway was the opposite: running these models is really hard, and the expertise involved is real and very specific.
Questions people ask
What does it mean to run a model?
Inference is the phase when a finished AI model answers live requests, as opposed to training, when it learns. Companies like Fireworks specialize in serving those answers quickly and cheaply for app developers.
What is the difference between speed and throughput?
Speed is how quickly one user gets an answer, while throughput is how many answers the same machines can handle at once. Throughput matters because it determines cost: higher throughput means cheaper answers and better margins, even if customers never see it directly.
Why would identical chips give such different results?
Software tuning, memory management, batching requests together, and custom networking can all affect performance. Small inefficiencies compound when a model has trillions of parameters and must shuttle huge amounts of data between chips.