Korva is a scroll of the internet’s best ideas.
Retry what the last model botched
Around Sonnet 3.5 or 3.6, Claude still made mistakes on side projects. Some engineers who were resisting AI tooling pointed to those mistakes as proof.
The harder thing to see was the exponential rate of improvement. Something tried before that Claude was not quite good enough at can be fine in the next model, and now it counts as a new capability.
The habit that follows is returning to what did not work, because the field moves fast enough that last year's dead end can be today's working feature.
Why do AI models improve in leaps?
A 2020 paper by Jared Kaplan and colleagues at OpenAI found that language model performance improves smoothly and predictably as model size, data, and compute grow. A smooth average can still hide sharp changes on particular tasks, which flip from failing to working as models scale. Users see this as a task that suddenly becomes possible. Each new model generation can therefore change which tasks are within reach.
How much have coding benchmarks changed?
SWE-bench, introduced by Princeton researchers in 2023, tests models on real GitHub issues from popular Python projects. In the original paper, the best model, Claude 2, resolved just under 2 percent of the issues. Later models resolved a far larger share. The benchmark has become a common way to track progress in coding ability.
More from Lenny's Podcast: Product | Growth | Career
