Korva is a scroll of the internet’s best ideas.
Human data work is all edge cases
Edge cases surface
A customer can write a guideline for how to make a data point, but the experts doing the work hit edge cases that bubble up. Handling them takes insanely fast alignment between customers, sometimes their customers, other experts in the field, and the people doing the annotation.
Operational intensity
It also takes a huge amount of paranoia from the operations team, to make sure every data point is perfect and fits the customer's guidelines, the project runs on time, and every bottleneck is removed. The work is operationally intense because it necessarily deals with things not seen before and outside model capabilities. The data types also change frequently, from supervised fine-tuning to preference ranking to rubric-based annotation to RL environments.
Moving down market
For labs, a human data project is a white-glove service that needs a lot of people on the operations team. That is why Mercor wants to move down market, with self-serve projects and AI project managers, so that smaller and more varied projects work for smaller customers. There are many more enterprises than labs, and that direction has already reduced revenue concentration.
How do teams measure annotator agreement?
A common measure is Cohen's kappa, introduced by statistician Jacob Cohen in 1960. It compares how often two annotators agree with how often they would agree by chance. A score near 1 means near-perfect agreement and a score near 0 means agreement no better than chance. Disagreements tend to cluster on borderline cases, which makes the score a way to find where a guideline is unclear.
How do self-serve products reach small customers?
Self-serve products let customers sign up and start without a salesperson, which makes small accounts affordable to serve. Atlassian built a large software business in its early years with almost no sales force, relying on customers finding and trying the product on their own. Customer support moves into documentation and automated onboarding.
Why is revenue concentration a concern?
Losing one customer that supplies a large share of revenue can cut a company's income sharply and quickly. US accounting standards require public companies to disclose any customer that makes up 10 percent or more of revenue, which lets investors see the exposure. Suppliers to a few large buyers also face pressure on prices, since the buyer can walk away.
More from 20VC with Harry Stebbings
