Korva is a scroll of the internet’s best ideas.
Grade AI by what users do
The copy-paste signal
OpenAI's chat product records whether a user copies and pastes a response. Hardly anyone taps thumbs up or thumbs down, so a copy is the usable signal: when a reply is good and gets pasted, the outcome was probably good. No paste is ambiguous, since it may or may not mean the reply failed.
Signals in other products
The same logic fits any product. A code review agent flags a P0 on a pull request, and if the PR is merged with no new commits pushed, the flag was probably wrong. A health chat gives an answer, and the patient either books a follow-up appointment or does not. Events like these are cheap, so every production run can be scored instead of a sample.
Why not a judge
The alternative is an LLM as judge, which means feeding a whole trajectory, maybe 10 sub-agents and 100 steps, into another model and asking whether it got the answer right. That is slow and expensive enough that most teams check a sample, use human reviewers, or skip production checks entirely. A product signal can also decide when a judge is worth running, such as reading a reviewer's feedback only after a PR is rejected.
Why do so few users tap thumbs up?
Rating a response takes a deliberate extra action that does nothing for the user's immediate goal. Explicit ratings also suffer from self-selection, since people with strong reactions rate far more often than people with mild ones. Implicit signals such as copies, clicks, and purchases are recorded for every user without asking for effort. Recommender-system research has long separated explicit feedback from implicit feedback for exactly this reason.
Why is an LLM judge slow and costly?
A judge model has to read the entire trajectory as input, and input is billed per token, so a run with many sub-agents and steps can mean a very large prompt for every single grade. The judge also runs after the work is finished, which adds latency. Its verdicts are imperfect too: a 2023 study of GPT-4 as a judge, published by Zheng and colleagues with the MT-Bench benchmark, found it agreed with human preferences more than 80% of the time.
How can a merged PR show a flag was wrong?
A reviewer flags a problem so that the author will fix it, and fixes arrive as new commits. When the pull request merges with no further commits, the author most likely saw nothing worth changing. Code review tools commonly track this as a resolution rate, the share of comments that led to a change in the code.
More from The Peel with Turner Novak
