Korva is a scroll of the internet’s best ideas.
Proactive agents need far more compute
Prompted versus proactive
Code generation tools work in a loop: the user prompts, the tool runs for a while, comes back, and waits for the next request. They do background work, but it follows the prompt. Instinct is closer to the reverse. Most of what it does happens without a prompt, and only a smaller share is interactive.
A morning wake-up
Picture an assistant that wakes at 6 AM because the user wakes at 7. It scans everything, gets the day ready, and decides that now is a bad moment for a notification, so it goes back to sleep or finishes one thing quietly.
Compute adds up
At 4 PM it sees something coming up that would help, wakes again, does the task, and contacts the user, who can pick it up from there. Each wake-up carries thinking about what could go wrong or right, and multiplied across a user base that adds up to orders of magnitude more compute than was planned.
Why does each wake-up cost so much?
Each time an agent wakes, the model has to read its notes, calendar, and recent events again before it can decide anything. The cost of processing text through a transformer model grows faster than the length of the text, because attention compares every word with every other word. A long context therefore costs much more than a short one. The attention mechanism was introduced in the 2017 paper Attention Is All You Need.
How does thinking add tokens?
Reasoning models write out a long chain of thought before giving an answer, and every word of it is a token that has to be generated. OpenAI released its o1 model in September 2024 with this design, and providers bill the reasoning tokens as output. A single hard question can produce many times more reasoning tokens than answer tokens. An agent that reasons at every wake-up repeats that cost each time.
Why do output tokens cost more than input?
A model can read all of the input at once, but it must generate its output one token at a time, with each step waiting on the one before. That sequential work uses hardware less efficiently than reading does. Major AI providers price output tokens at several times the rate of input tokens for this reason.
More from Invest Like the Best with Patrick O'Shaughnessy
