Korva is a scroll of the internet’s best ideas.
Same model, different results by host
Moonshot recently posted a benchmark showing how well each inference provider serves Kimi K3, and the numbers differ even on benchmarks that are static and well known. OpenRouter benchmarks all of the models on all of the open weight providers continuously, and the results keep changing. These models are very non-deterministic.
When one provider makes a token go further, with a quality improvement, a speed-up, or a price reduction, OpenRouter's routing sends it more traffic right away. For the big models there are big changes every five minutes.
Providers like Fireworks and Together host open weight models better and faster than the hyperscalers. GLM on a hyperscaler is something almost nobody runs.
Why would one model score differently by host?
Providers can serve the same weights at different numeric precision, a technique called quantization that saves memory and money but can slightly change outputs. They also use different inference software, sampling defaults, and prompt templates. Any of these can shift the scores a benchmark reports.
Why does the same prompt give different answers?
Floating-point addition gives slightly different results depending on the order of operations, and GPUs reorder them as they split work across many cores. Thinking Machines Lab published a 2025 analysis, Defeating Nondeterminism in LLM Inference, that traced much of the variation to how requests are batched together on a server. Tiny differences in early tokens can then compound into different answers.
Why do providers differ in speed?
Providers run on different hardware, from several generations of Nvidia GPUs to custom chips such as those made by Groq and Cerebras. They also use different serving software and different ways of splitting a model across chips. Speed per token can therefore vary several times over for the same model.
More from 20VC with Harry Stebbings
