Korva is a scroll of the internet’s best ideas.
Cheap everyday use
The vast majority of people using AI today use it as a Google substitute or a recipe maker, and the cost to serve them is extremely low, similar to serving a web page and only marginally higher.
Thinking for days
At the other extreme are people using test time scaling. Models used to improve by getting bigger and bigger. Now they can improve by thinking longer about the answer, for days, weeks, or months, and every second of thinking costs more money.
Costs vary widely
Inference costs are real, and far more real than in the prior era, when computers ran the ad purchases at Google and Meta at essentially $0. The user on free ChatGPT and the user trying to solve a math theorem are not even remotely in the same universe, and gross margins vary by company with an incredible spread.
How does test-time scaling work?
A reasoning model writes out a long chain of intermediate steps before it gives its answer. Each of those steps is a token that takes computation, so thinking longer costs more. OpenAI's o1 preview, released in September 2024, brought this approach into wide use.
Why does a longer thought cost more per second?
In a standard transformer model, each new token looks back at all the tokens before it, and the model stores that history in memory. The longer the thought, the more history it must keep and read. The attention computation in these models grows with the square of the sequence length.
Why is serving a recipe so cheap?
Serving a web page mostly means looking up stored content and sending it. A language model instead runs billions of parameters for every token it writes. A short answer finishes in seconds, so the total stays small even though each token has a real cost.
More from Invest Like the Best with Patrick O'Shaughnessy
