Korva is a scroll of the internet’s best ideas.
Test the cheaper model first
The check is the same as grading by product outcomes. Roll the new model out to a slice of users, track the product signals, and see whether the results match while token costs fall. The hope is a cut of 10 to 20 times, though that is something to measure, not assume.
Over time this builds a data set. With product signals marking which agent trajectories ended well, a team has known good trajectories it can use for post-training an open-weights model.
The result is a smaller model fitted to one product, which can be swapped in through the same experiment method. Frontier models are very capable but generic and expensive, while a model trained on a product's own successful runs is cheaper and more efficient to run.
How do successful runs train a smaller model?
Runs that ended well are kept as examples, and a smaller model is fine-tuned to imitate the reasoning and tool calls in them. Filtering by outcome and training only on the winners is sometimes called rejection sampling. Teaching a small model from a larger model's outputs is known as distillation, which Hinton, Vinyals, and Dean described in a 2015 paper.
Why can open-weights models cost less to run?
Several inference providers host the same open weights and compete on the price per million tokens, and a team can also run them on its own hardware. Smaller models need less GPU memory and compute for each token generated. Mixture-of-experts designs go further: DeepSeek-V3 activates about 37 billion of its 671 billion parameters for each token.
Why measure savings instead of assuming them?
A lower price per token does not guarantee a lower price per finished task. A weaker model may take more steps or retry more often, and each extra step adds tokens. Agent loops also reread a growing context at every turn, so total tokens over the whole run drive the bill.
More from The Peel with Turner Novak
