Korva is a scroll of the internet’s best ideas.
The fraud exchange
A recurring exchange in growth meetings: someone dismisses a result as fraud. How do you know? It is labeled fraud in the dataset. Who labels it? Someone in ops. What is their procedure? Nobody knows, so nobody knows it is fraud.
Seasoned leaders dig in
Push a seasoned product director on this and they find out. They end up strengthening the system that agents use to label data, because a very smart person is now invested in how it works and helps guide it.
Why weeds everywhere
That is the case for pushing the best people into the weeds everywhere: fewer things get done, and they get executed thoroughly, and lots of problems get fixed along the way. Growth hides every sin otherwise, because a bigger company just scales up and everyone works off averages.
How often are dataset labels simply wrong?
In 2021 researchers Curtis Northcutt, Anish Athalye, and Jonas Mueller studied ten widely used machine learning benchmark datasets. They estimated that the test sets contained label errors at an average rate of about 3.4 percent. Several of those datasets are standard references in image and text research. A label records what a person decided at the time, which makes it a human judgment.
Why can fraud labels arrive months late?
In card payments, a cardholder can dispute a charge through a chargeback, and the networks allow disputes for a long window, commonly up to 120 days. A transaction that looks fine on the day it happens can be relabeled as fraud months afterward. Datasets built from recent transactions therefore undercount fraud for recent periods. Analysts correct for this by waiting for the window to close before treating labels as final.
More from Lenny's Podcast: Product | Growth | Career
