Korva is a scroll of the internet’s best ideas.
Agent companies will build models
In July, OpenRouter launched 70 models, about one every 10 hours. Cognition has a model and Cursor has a model. Lovable doesn't have one publicly.
Several pressures push more founders toward new labs: Nvidia wants more competition and diversity in the space, and investors want to try new things that could improve intelligence in some different way.
If American open weight models pick up steam, they give these neo labs a base to train on that isn't Chinese, which will probably create more American neo labs.
Why does agent usage data help train a model?
Every edit a user accepts or rejects in an agent is a signal about which output worked. OpenAI's 2022 InstructGPT paper showed how human preference signals can tune a model, and labelers preferred outputs from a 1.3 billion parameter InstructGPT model over those of the 175 billion parameter GPT-3. A company running an agent collects such signals as a byproduct of normal use.
How cheaply can a company build on an open base model?
Starting from published weights skips pretraining, the most expensive stage. Meta's Llama 2 paper reported pretraining on 2 trillion tokens, an effort that took months on thousands of GPUs. Fine-tuning a released model on domain data uses a small fraction of that compute.
Why would a company that pays for models build its own?
Companies that resell access to a lab's model pay that lab for every token their users generate. A model the company owns can be served at its own cost, so the margin does not depend on a supplier's pricing. It can also be tuned to the product's exact tasks.