Korva is a scroll of the internet’s best ideas.
The jagged edge
The jagged edge is the uneven shape of what models can do. Human skill climbs in a smooth arc, and a model's doesn't: it is brilliant in some places and falls short in others, and the shape changes every few weeks. The strongest AI application companies work close to the models, learn where they fail, and build around those gaps.
Product work inverted
That inverts traditional product management. The old job was to understand the customer and hand the problem to engineers, without specifying how to solve it. Now the work requires knowing what the models are great at and where they break, and joining that with the customer's problem to bridge the gap.
Titles matter little
With those three things, the job title matters little. An engineer, a product manager, or a designer who has taste, understands the customer's problem, and is curious about the jagged edge does well.
Why is AI capability so uneven?
A model is strong where its training data and feedback are plentiful and weaker where they are thin, so skill does not rise evenly across tasks. Harvard Business School researchers working with Boston Consulting Group consultants called this a jagged technological frontier in a 2023 study. In it, consultants using GPT-4 did better on tasks inside the frontier and worse on a task outside it. The study involved 758 consultants.
Where did product management come from?
In 1931, Neil McElroy of Procter & Gamble wrote a memo proposing that each brand have a dedicated manager responsible for it. That brand-manager role became an ancestor of product management. Technology companies later adapted it, with product managers sitting between customers and engineers. McElroy later served as US Secretary of Defense.
How do teams find where a model fails?
They build evaluations, sets of test tasks drawn from the real work, and score each new model on them. Public benchmarks give a rough picture, but a team's own tests show how a model fails on its particular problem. When a new model arrives, rerunning the evaluations shows what improved and what broke. MMLU, a widely used benchmark published in 2020, covers 57 subjects.
More from Invest Like the Best with Patrick O'Shaughnessy
