Korva is a scroll of the internet’s best ideas.
Environments are simulated worlds for agents
What an environment is
An environment is a simulation of an app that an agent might need to use, paired with a rich start state: a world that represents all the data you might have on your machine, anywhere from hundreds to thousands of files. Tasks then train the agent to use those tools to accomplish something useful. The annotation is complicated because the agent has to interact with the simulated world.
Why fidelity matters
The data agents are now evaluated and trained on looks a lot closer to what they meet in deployment. An agent that needs to learn Salesforce needs a high-fidelity mock that acts exactly like Salesforce. Setting that up is hard, the way preference ranking and supervised fine-tuning were hard to set up when InstructGPT first came out, and labs are still figuring it out. Eventually it should get smooth enough for enterprises to do it too.
Growing demand
Environments are the fastest-growing data type at Mercor, which counts itself the leader in the category. Last year, delivering them was so complicated that scaling them looked doubtful. Demand was very high, the team tried a lot of different things to make environments actually improve model performance, and it worked.
Why do agents train in simulated apps?
Reinforcement learning works by trial and error, and an agent practicing on a live system could delete records, hit rate limits, or leave a mess it cannot undo. A simulation can be reset to an identical start state thousands of times. It can also check the result automatically, such as whether the right field in a record changed. OpenAI released the Gym library in 2016 to standardize this kind of environment for research.
How did game environments lead to these?
DeepMind's 2013 work on Atari games let a program learn from the screen alone, using the game score as its reward. Games make good training grounds because the score gives instant, unambiguous feedback. Office software gives no score, so builders must write checks for what counts as success. The Arcade Learning Environment, published by University of Alberta researchers in 2013, packaged dozens of Atari games for this kind of research.
Why must a mock behave exactly like the real app?
An agent learns whatever the simulation rewards, so any difference between the mock and the real app can teach habits that fail in deployment. Researchers call this the sim-to-real gap, a term from robotics, where policies trained in simulated physics often fail on real machines. Close copies reduce the gap but cost more to build and maintain. Robotics researchers have used domain randomization, varying the simulated conditions on purpose, to narrow it.
More from 20VC with Harry Stebbings
