AI Labs Can Study for Old Questions, but New Ones Keep Coming
LMArena co-founder Ion Stoica says at one point most prompts were new, driven by news and by smarter models.
Lightspeed Venture Partners · Meta Can Read Your Brain, The AI Science Race & AI Ranking With Ion Stoica | LightworkThe worry was familiar. If a scoreboard drives the industry, labs will study for it, the way students study for standardized tests. Ion Stoica, a Berkeley professor and co-founder of LMArena, the live leaderboard where people pit anonymous AI models against each other and vote, had just been asked how he picks the right measurement given the leaderboard's influence over the whole industry.
He says: "You can optimize maybe for the past task but they are going to see newer and newer tasks."
By that, he means a live test works differently from a fixed exam. A fixed set of questions can be memorized. His system is dynamic, fed every day by real users bringing whatever they actually need that day.
The churn is large. He pointed to an internal study, which he cautioned not to quote as exact, that at one point almost 70% of user prompts had not appeared in the prior three months. That is, most user prompts were new at that time.
The reasons he gave were twofold. One is the world itself. People ask about what happened today, the news and events that did not exist three months ago. The other is the models themselves. New models arrive every few weeks or so with new capabilities, so users attempt bigger and harder tasks than they could before.
So a lab can tune its model to do well on last month's questions, but that edge fades as new prompts keep arriving.
Questions people ask
How does LMArena actually rank models?
Visitors type in a real question and get answers from two unnamed models side by side. They vote for the better answer, tie them, or discard both, and those votes feed a chess-style rating where beating a strong opponent counts more than beating a weak one.
What does it mean to overfit to a test?
It means tuning a system to do well on known questions rather than learning the underlying skill. It is like memorizing last year's exam instead of learning the subject, so performance drops when fresh questions appear.
Why would AI labs care about topping a public leaderboard?
A high rank signals quality to developers, businesses, and investors choosing which model to build on. That attention can shape partnerships, usage, and reputation, which is why Stoica says trust in the referee is the business itself.