Korva is a scroll of the internet’s best ideas.
Not everything is urgent
Calling a frontier model's API from a major provider means paying a blanket cost on every request, including the ones that need answers in hundreds of milliseconds and the ones that don't. A proactive assistant does plenty of the second kind, such as consuming large amounts of content in the background. Those jobs have no reason to finish instantly.
Shaped deployments
Instinct shapes its inference deployments around the data and workloads it serves. Batch work runs on setups that are three to eight times more efficient on the same compute. Other tuning finds 30 percent in one place and five or six times in another, and the savings compound.
The measured result
On Instinct's own A/B tests, internal evaluations, and engagement tracking across users, the result matches Opus 5, a frontier-level model, at a very low cost. That cost is what lets every user have the product for free.
Why is batch processing cheaper?
A GPU is most efficient when it processes many requests together, because the cost of loading a model's weights from memory is shared across all of them. Waiting to collect a larger batch raises throughput but adds delay for each request. Both OpenAI and Anthropic launched batch interfaces in 2024 that return results within hours at a 50 percent discount. The discount reflects the fact that the provider can run those jobs whenever its hardware is idle.
How do labs speed up AI responses?
Engineers use several techniques to get more output from the same chips. Quantization stores a model's numbers at lower precision so they take less memory. Speculative decoding has a small, fast model draft several words that the large model then checks all at once. Google researchers described speculative decoding in 2022 and reported generation that was two to three times faster with identical outputs.
What does a key-value cache save?
When a model generates text, it computes internal values for every earlier word each time it adds a new one. A key-value cache stores those values so they are computed once and reused. Without it, the cost of generating each word would grow with the length of everything before it. Caching is a standard part of nearly every system that serves language models.
More from Invest Like the Best with Patrick O'Shaughnessy
