Korva is a scroll of the internet’s best ideas.
Same model, five times faster
AWS, Azure, GCP, the neoclouds, and Fireworks all run the same stock open-source models on the same Nvidia hardware. Fireworks delivers about five times the speed of a cloud provider on that model. The throughput gap is a multiple as well, and it shows only in the economics of the business, since throughput is invisible from outside.
Those companies pay the margins of the cloud providers, run on top of them, and still make money. Running models with two, three, or four trillion parameters efficiently is very hard, and it takes a very specific expertise.
Seen from the outside, inference looks like a commodity, a pass-through resale business won by scale, the same way the early days of AWS looked to investors. Up close, it is anything but.
Why is generating text bound by memory?
Each new token a model produces requires reading all of the model's weights from GPU memory. The arithmetic per token is small compared with the cost of moving those weights. This is why memory bandwidth, not raw compute, often sets the speed of text generation. A model with trillions of parameters has to spread across many GPUs, which adds communication between them.
How does speculative decoding speed up models?
A small, fast model drafts several tokens ahead, and the large model checks them all in a single pass. Tokens that match are kept, and the first wrong one is replaced with the large model's own choice. Researchers at Google described the method in a 2022 paper called Fast Inference from Transformers via Speculative Decoding. The output is the same as the large model would have produced alone.
How does batching raise throughput?
A GPU finishes one request far below its capacity, so serving many requests together uses the hardware better. The Orca system, presented at a 2022 systems conference, scheduled batches at each token step so new requests could join while others were still running. The vLLM project later introduced PagedAttention in 2023, which cut memory waste in the cache that stores earlier tokens. Both ideas raise how many requests a single GPU serves per second.
More from Invest Like the Best with Patrick O'Shaughnessy
