Korva is a scroll of the internet’s best ideas.
The speed gap
The models in question have two to four trillion parameters, and running them is brutally hard. Amazon, Microsoft, Google, the newer GPU clouds, and Fireworks all serve the same stock open-source models to developers, which makes the comparison clean. Against a cloud provider, Fireworks is about five times faster.
The hidden half
Speed is only the visible half. Throughput, how much work the hardware gets through, can't be seen from outside and shows up only in the economics of these businesses. There the gap is a multiple too. Companies like Fireworks pay the cloud providers' margins, run on top of their hardware, and still make money.
Not a commodity
From the outside, serving models looks like a commodity: pass-through resale where scale decides everything. The numbers point the other way. Running these models efficiently takes a narrow kind of expertise, and not many teams have it.
Why can identical chips serve models at different speeds?
Serving speed depends heavily on the software around the chip. At this scale, generating each token is often limited by how fast the chip can read the model's weights from memory, so how the work is scheduled and batched matters as much as raw compute. Tricks such as speculative decoding, in which a small model drafts tokens that the large model checks in parallel, speed it further. Kernel code written by hand for a specific chip can also change throughput substantially.
Why is a trillion-parameter model hard to serve?
At one byte per parameter, a trillion-parameter model needs about a terabyte of memory for its weights alone. The model has to be split across many chips, which then spend time exchanging intermediate results over fast links. Keeping every chip busy while that traffic flows is the central engineering problem. A single NVIDIA H100 GPU carries 80 gigabytes of memory.
Why does utilization decide serving costs?
A GPU costs the same to rent or own whether it sits idle or runs flat out. Serving many users' requests together on one GPU, called batching, spreads that fixed cost across more output. The more tokens a chip produces per hour, the lower the cost of each one. Providers therefore compete on how many tokens per second each GPU delivers.
More from Invest Like the Best with Patrick O'Shaughnessy
