Korva is a scroll of the internet’s best ideas.
Margin moves to infrastructure
One token index dipped and flattened last month, and the market took it as bad news. The index captures mix, and it showed a shift from expensive frontier tokens, which probably carry inference margins of 80, 90, or even 95 percent, toward open-source tokens from models like GLM 5.2 and Kimi.
Open-source tokens run on the same cloud providers as the frontier ones, and those providers charge the same for compute. So the shift pulls margin dollars out of the frontier model layer and pushes them into AI infrastructure. Cheaper tokens also stretch demand, because there is elasticity, and that means more demand for compute.
Routers at work
Companies that route work between models show the same effect. Some big public companies saw their AI spend grow 20 times and burned through a budget in three months. A router sends each request first to a customized open-source model and then to a frontier model that checks it, which in many cases gives slightly better outcomes at half the cost.
That saving is tokens moving from roughly 90 percent gross margin to roughly 30. The cheaper tokens may even raise the number of tokens a company generates, which means more GPU hours behind the router.
Why do cheaper tokens raise total demand?
Economists call this the Jevons paradox. In 1865, William Stanley Jevons observed that more efficient steam engines had increased Britain's coal consumption, because cheaper power opened up new uses. When a resource gets cheaper, people find more things to do with it. The same pattern has shown up in lighting, where falling costs over the centuries led to vastly more light being used.
Why are open-source model tokens cheaper?
Open-weight models are files of numbers that anyone can download and run on rented servers. Many hosting companies compete to serve the same model, and in a competitive market prices tend to settle near the cost of production. A frontier lab that is the only seller of its model has no such competitors, which leaves room for much fatter margins. The hardware underneath costs the same either way, since a GPU hour is billed at the same rate whichever model runs on it.
How does a model router save money?
A router is a small program that decides which model should handle each request. Easy requests go to a cheap model, and hard ones escalate to a stronger one. A 2024 paper from researchers at LMSYS and Berkeley, called RouteLLM, reported cost reductions of more than 85 percent on one benchmark while keeping most of GPT-4's quality. The savings come from the fact that most requests in real traffic are easier than the hardest ones.
More from Invest Like the Best with Patrick O'Shaughnessy
