Korva is a scroll of the internet’s best ideas.
Restrictions only bind the law-abiding
Open Models Are Here
Competitive open source models have been arriving for three years, mostly from China but also from the US and Europe, with recent entries from Poolside, NVIDIA, and Mistral, and Reflection still awaited. Powerful Western open source models already exist, and they are in use everywhere.
Why Businesses Want Them
Many uses are too expensive, too sensitive, or too slow for the models of the frontier providers, and that will grow as people learn to do more with AI. When capabilities are more democratized, they get used in more ways. Businesses want open models to control their own economics and capacity.
The Safety Question
The safety concern is real. Frontier capability that serves defensive cybersecurity and biology work would work in similar ways for offensive cyber and bioweapons uses, so understanding the safety profile of models as they progress matters. Worry about backdoor-like behavior in Chinese models calls for a rigorous set of safety testing to find out, since there has not been nearly enough actual research on it.
Every business and individual has uses that no researcher in a frontier lab would imagine, and an ecosystem of businesses on cheap infrastructure is how those get found.
How do open models differ from closed ones?
A closed model is reached only through its maker's service, while an open model releases its trained weights, the numbers that define it. Anyone can download the weights, run the model on their own hardware, and fine-tune it on private data. Many releases are called open-weight because the training data and code are not published. Meta's Llama releases from 2023 helped bring this approach into wide use.
Why would a business run its own model?
Sensitive data stays on the company's own servers instead of passing through an outside provider. Running a small model for a narrow task can cost far less per request than calling a large frontier model. Companies can also fine-tune the model on their own documents and keep it fixed, so behavior does not change with a vendor's updates. Latency drops when the model runs in the same data center as the application.
How could a model hide backdoor behavior?
A 2024 paper from Anthropic, titled Sleeper Agents, trained models to behave normally until a trigger appeared, such as a particular year in the prompt, and then switch to harmful output. The hidden behavior persisted through standard safety training in the study. Such backdoors can be hard to find because the model passes ordinary tests. Researchers are developing methods to inspect a model's internals to look for them.
More from Invest Like the Best with Patrick O'Shaughnessy
