Korva is a scroll of the internet’s best ideas.
You can't overfit a moving benchmark
Anything that gets measured gets targeted, and labs do adapt to be higher on a leaderboard. A fixed set of benchmarks can be optimized against, but a live system cannot.
People ask about what happened today, so the prompts follow the news. A new model arrives every few weeks, and as capabilities advance, people try bigger and harder tasks than they could before. Studies at Google also found that there are always new queries.
A lab can optimize for past tasks, but newer ones keep appearing. For that reason the whole system is dynamic and tests live, with everything changing every day.
What does overfitting to a benchmark mean?
Overfitting happens when a model or its builder tunes so closely to a fixed test that scores stop reflecting general ability. It can occur directly, by training on the test questions, or indirectly, by repeatedly picking whichever version scores best. Economist Charles Goodhart's law states that a measure used as a target stops being a good measure. Image recognition competitions in the 2010s saw leaderboard gains that shrank on fresh data.
How do new prompts keep a test fresh?
Because users write prompts about that day's events and projects, the test set is never the same twice. A lab cannot train on questions that have not been written yet. Public data sets, by contrast, get scraped into training corpora over time. Live sampling gives a moving target that tracks what people actually ask.
Why do people attempt harder tasks over time?
Once a tool handles a task reliably, users move on to more ambitious ones. When early chatbots could write a short poem, people began asking for whole programs. The mix of requests therefore drifts toward the frontier of what models can do. Query logs at search engines show a similar pattern, where a notable share of daily searches have never been seen before.