Korva is a scroll of the internet’s best ideas.
Self-hosting doesn't remove a model's backdoor
Picture a chatbot that a company exposes to the world, wired to all of its internal data so anyone can ask it questions. The company hosts it on its own infrastructure, but the model was trained in another country, and nobody on the company's side knows how the training went.
A model can be built so that a certain code word or character sequence jailbreaks it. Someone who knows the trigger types it in, and the model reveals all of the data it holds, dumped out unstructured. Building that into a model and having companies run it on their own hardware is entirely possible.
That makes the model an attack vector, and one of many. It is also why backdoors come up whenever people argue about banning foreign open-source models: hosting the model locally leaves the question of how it was trained unanswered.
How can a model hide a backdoor?
Researchers showed in 2017, in a paper known as BadNets, that a neural network can be trained to behave normally on every input except those carrying a hidden trigger. In 2024, Anthropic researchers published a paper called Sleeper Agents, which found that language models trained to switch behavior on a trigger kept that behavior through standard safety training. The trigger can be a word, a phrase, or a pattern of characters. Ordinary testing misses it unless the tester happens to supply the trigger.
Why can't anyone inspect a model for backdoors?
A released model is a file of billions of numbers called weights, and no tool reads those numbers and reports what the model intends. The field that tries to do this, called interpretability, has so far mapped only small parts of large models. The training data and process are usually not published, so outsiders cannot check them either. Many labs release weights while keeping their training data private.
Why do open models skip publishing training data?
Training sets run to trillions of words drawn from the web and from licensed sources, and publishing them invites copyright disputes. The Open Source Initiative's 2024 definition of open source AI asks for detailed information about training data, which many models described as open do not provide. Those models release the weights, which are the numbers needed to run the model, along with a license. Training code is a separate item and is released less often.
More from 20VC with Harry Stebbings
