Korva is a scroll of the internet’s best ideas.
Students fine-tuned Llama in a weekend
A weekend fine-tune
They put Vicuna on the web as a chatbot and faced a problem: how do you show it is better than the base Llama? Static tests like question-answering benchmarks did not capture how real people talk to language models.
Building the Arena
So they built a comparison tool they called battle mode. A user asks any question they like, two randomly chosen models answer, and the user picks the better one, or calls it a tie, or rejects both. They named it Arena.
For a long time it was a side measurement project that nobody paid much attention to. Then it gained traction, and large model providers began pointing to their Arena results to show how well they were doing, because it used real users on real tasks.
Paying for neutrality
Running it was costly. Arena pays for access to the best models so users can use them free, and the team needed to stay neutral rather than rely on free credits from model makers. They had wanted to donate it to a foundation, but the numbers did not add up, so they made it a company.
Why did students pick a weekend project like this?
Fine-tuning takes a finished base model and trains it a little more on a small, targeted data set, which costs far less than training from scratch. Releasing Llama's weights meant outside researchers could do this at all, since earlier top models were available only through a company's own service. Stanford's Alpaca project, released around the same time, used the same recipe of tuning Llama on conversations generated by another model. That pattern set off a wave of open chatbots built on Llama.
Why can't fixed benchmarks capture real chat?
Benchmarks such as multiple-choice question sets have a single right answer and a fixed list of questions. Real users ask open-ended things with no answer key, like advice on an email or a story idea. Once a benchmark is public, its questions can also leak into training data, which inflates scores. Researchers call this contamination, and it is a long-running problem in language model evaluation.
Why let users vote without knowing the models?
Hiding the model names removes brand bias, since people tend to favor a name they already trust. Pairwise comparison is also easier for humans than scoring one answer on a scale, because people disagree far more about absolute scores than about which of two is better. Pairwise votes feed naturally into rating systems built for head-to-head results. Psychologists have long observed that people judge comparisons more reliably than absolutes.
More from Lightspeed Venture Partners
