KorvaJoin the waitlist
Tech

Remembered Tokens Cost Up to 100 Times Less Than New Ones

For customers with large requests, missing that memory changes what they pay.

AI image
Standard Capital · Luminal Raises $25M Series A to Break AI's Hardware Bottleneck, Led by Standard Capital

A token an artificial intelligence system remembers can cost up to 100 times less than a token it processes for the first time. A token is a small piece of a word, the unit these systems count.

The savings come from a kind of short-term memory called the KV cache. It stores math the system already did, so its costly chips do not have to do the work again.

For Freebuff, which offers free AI use on its platform and pays Luminal for the work behind it, that matters. Freebuff sends Luminal trillions of tokens a week, and a single request can carry around 100,000 input tokens, so even a small drop in finding the right memory changes what it pays.

Luminal says it offered pricing competitive with the official DeepSeek API. Handling that cache was one of several reasons it could do so while using far less hardware, alongside work on its compiler and other scaling problems.

Questions people ask

Why does an AI system need to remember words at all?

When a chatbot writes an answer, it looks back at all the earlier words each time it adds a new one. Without memory, it would redo all that math for every word, which takes time and chip power. The KV cache saves those earlier calculations so the system only does the new work.

Why would one request include 100,000 input tokens?

A coding assistant often reads a large amount of existing code, chat history, and instructions before it suggests a fix. That context can run to tens of thousands of words, which the model counts as input tokens. If much of that context repeats from one request to the next, remembering it avoids paying to process it again.

What makes a cache miss so expensive?

A miss means the system must run the full calculation on costly, power-hungry chips. The chips take longer, use more electricity, and can serve fewer customers at once. At trillions of tokens a week, those extra costs add up quickly.

More in Tech