Korva is a scroll of the internet’s best ideas.
Replay step 18, change it
Replaying one step
Recording what happened in a 20-step run is not the same as being able to replay it. Without deterministic playback, rerunning the first 18 steps and altering the last one is particularly hard. When the infrastructure that runs the steps also creates the traces, it holds far more context, and that speeds up building an agent that improves itself.
Testing on a slice
The same setup makes A/B testing simple. The infrastructure can fork two different steps or groups of steps, send a change in prompt, model, or harness to 1% of users, and compare results. A swap between Anthropic and OpenAI could go to 50% or 20% of users, with the check being that outcomes match while token costs drop.
The usual practice
The usual practice is different: change a prompt locally, run evals on maybe 100 test cases, and if it looks good, push it to every user. Spot checks of production afterward use an LLM as judge or human review of a sample.
Why is replaying an agent run so hard?
Language models sample their output with randomness, and even at a temperature of zero, results can differ between runs because of floating-point arithmetic and how requests are batched on the hardware. Tools add more variation, since a web search or database query returns whatever is true at that moment. A replay system sidesteps this by storing each step's result and returning the stored value instead of running the step again. Event sourcing, a long-standing pattern in databases, rebuilds state from a recorded log in the same way.
How does a 1% rollout differ from offline evals?
Offline evals run a fixed set of test cases, which may not look like real user traffic. A small live rollout sends a slice of actual users to the new version while everyone else stays on the old one, so both groups face the same conditions at the same time. This pattern is called a canary release, named for the canaries once carried into coal mines. It lets a bad change reach only a small number of people before it is rolled back.
Why does the run's location matter for traces?
A trace recorded by a separate monitoring tool sees only what the application chooses to report. Infrastructure that actually executes each step sees every input, output, retry, and timing as it happens. That fuller record is what makes it possible to restart from a middle step with identical earlier results. Distributed tracing systems built on the Dapper paper, published by Google in 2010, exist because partial records of this kind were hard to debug.
More from The Peel with Turner Novak
