Korva is a scroll of the internet’s best ideas.
Judging agents by their traces
An agent is an orchestrator of language models that can interact with its environment: it uses tools, searches, browsers, and databases, and a harness gives it memory and keeps its state. A task like buying a blue cotton sweater within a price range takes many rounds of back and forth.
Arena takes that trace and splits it into its individual interactions. Inside it are signals: a user says something is good or bad, asks the agent to do it again, or says to move on to the next step. More explicit signals include a task marked complete or a user giving up on a clueless agent.
From those signals Arena can compare different combinations of model, tools, and harness. This is the first time agent capabilities are being evaluated on real tasks from real users.
Why is judging an agent harder than a chatbot?
A chatbot produces one reply that a person can read and rate. An agent takes many steps, and an early mistake can quietly ruin the final result. Two agents can reach the same answer by routes of very different cost, speed, and reliability. Researchers have built agent benchmarks such as SWE-bench, which checks whether an agent's code change passes a project's real tests.
What is a harness around a model?
A harness is the software wrapped around a language model that decides what the model sees and what it can do. It stores earlier steps, passes tool results back in, and limits how long the loop runs. Two harnesses built on the same model can behave very differently. Choices such as how much history to include affect cost, since models process text in units called tokens.
Why do language models need tools at all?
A language model only produces text, and its knowledge stops at the date its training data ended. Tools let it look up current information, run calculations, and take actions in software. Calling a calculator, for instance, avoids the arithmetic mistakes models sometimes make. A 2023 paper from Meta called Toolformer showed that models can learn to decide for themselves when to call such tools.
More from Lightspeed Venture Partners
