Why long chats make AI lose the thread
The chatbot's short-term notes on your chat can grow larger than the model itself, and it was mostly trained on much shorter talks.
Invest Like the Best with Patrick O'Shaughnessy · Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]Talk to an AI long enough and it starts to lose the thread. It is not your imagination. The notes it keeps on your conversation can grow larger than its brain.
That is how Neil Movva, founder of the AI infrastructure company Sale Research, explains it. A chatbot has two kinds of memory. One is its weights, the crystallized knowledge baked in during training. The other is what engineers call the KV cache, "the dynamic knowledge of the exact conversation" you are having right now.
Each token you feed into the model leaves a representation in that cache. A token is a small fragment of a word, the basic unit of text the model reads and writes. Have a 100,000-token back and forth, and when the model predicts token 100,001, it is still referencing everything behind it. Store a representation for every piece, and that pile of short-term memory frequently gets larger than the weights themselves.
That size is only half the problem. The model was mostly taught to think with much less short-term memory.
Models are trained primarily on shorter passages, around 8,000 to 16,000 tokens at a time, Movva says. They get some training at 200,000 tokens, but that is not their core strength. So keeping a model as smart at 200,000 tokens as it is at 10,000 has become a constant battle for the leading labs.
Even million-token context windows, meaning windows where the model can see that much text at once, often do not work well at full length. Movva says he still uses a command to compact his own coding sessions well before the one-million limit, because he does not think it works well near the full length.
The deeper inefficiency, he argues, is that the cache is bloated. The system now stores many kilobytes of data per token, far more than the information seems to warrant, perhaps by a factor of ten or a hundred. He points to published work from DeepSeek that, in his telling, compresses it by roughly an order of magnitude every year or so, which suggests how much fat is left to cut.
For now, he calls that growing memory a big thorn in the side of fast designs, and making models as smart across long chats as short ones a perennial battle.
Questions people ask
Why not just delete the start of a long conversation?
You can, and many tools do condense or restart chats for exactly this reason. The trade-off is that the model loses details it may need later, like names, decisions, or instructions from earlier. Compression tries to keep the meaning while shrinking the memory.
How big is this memory in practice?
At several kilobytes per token, 100,000 tokens means hundreds of megabytes. At a million tokens, that becomes several gigabytes. That is why long chats demand chips with lots of fast memory.
Will new chips fix long conversations?
Partly, but not alone. Chips that keep everything on board are very fast for the model's fixed knowledge, yet they struggle with a growing conversation memory. The likely fix is a hybrid: one kind of chip for the model's weights, and a more conventional chip with large outside memory for the conversation.