AI's most-watched ranking hides the names and lets users vote
Wins against top models count more, just as in chess and tennis.
Lightspeed Venture Partners · Meta Can Read Your Brain, The AI Science Race & AI Ranking With Ion Stoica | LightworkTo rank chatbots, the most watched scoreboard in artificial intelligence does not give them a quiz. It asks you to bring your own homework.
A visitor to LMArena, the platform that began as Chatbot Arena, types in a real task or question, anything from homework and computer code to dinner ideas. The site returns two answers side by side, from two randomly chosen AI models whose names are hidden.
The voter reads both and picks a winner, or calls it a tie. Voting is optional, but many people vote because they want to contribute, said Ion Stoica, a Berkeley professor and co-founder of LMArena.
Each vote is a tiny match. LMArena adds up millions of such matches into a single number for each model, using a system modeled on the Elo ratings used in chess and the tour rankings used in professional tennis.
The key is that not all wins count the same. Beating a strong opponent lifts a model's rating a lot, while beating a weak opponent lifts it only a little, and losing works the same way in reverse.
That weighting matters because models do not all face the same rivals. Random pairings mean a newcomer might face a champion one day and a lesser model the next, but the math adjusts for the strength of the opponent.
The result is a leaderboard that reflects what ordinary users found most useful in daily work, rather than performance on a fixed, standardized test. In a field where capabilities change every few weeks, the ranking moves as fast as people's questions do.
Questions people ask
Why hide the model names?
To keep the vote about quality, not reputation. If voters saw that one answer came from a famous lab and the other from an unknown startup, many would favor the famous name before reading carefully. Hiding the names forces a blind comparison.
What exactly is an Elo rating?
Elo is a scoring formula invented for chess by physicist Arpad Elo. Each player has a rating, and after every game points move from loser to winner, with more points moving when an underdog wins. It is now used in many sports and games because it can rank large groups that never all play each other.
How is this different from a standardized AI benchmark?
A fixed test uses the same questions for every model, so labs can study the test and train specifically for it. A live vote uses fresh questions from real users, so a model has to handle problems it has never seen. The tradeoff is that human votes can be noisy and reflect style as well as correctness.