Korva is a scroll of the internet’s best ideas.
The refusal
The model's guardrails kicked in because it could not tell what Hugging Face was doing and thought the company might be using the capabilities nefariously. A request from a defender looked no different from one an attacker would make.
A French company, a Chinese model
Hugging Face is a French company, and it is about to be acquired by NVIDIA. A company that is part of the American AI ecosystem was forced to turn to a Chinese model for cyber defense because the guardrails kept the latest American models out of its hands.
Who guardrails hurt
Guardrails do not stop bad actors from getting these capabilities, since they will have them anyway. The effect lands on defenders, who then lack the newest tools in the race to find vulnerabilities and patch them.
Why do AI models refuse security requests?
Cybersecurity knowledge is dual-use: the steps to test a system for flaws are the same steps used to break into it. A model has no reliable way to verify who is asking or why, so labs train models to decline requests that resemble attack techniques. Those refusals are tuned to avoid the worst misuse, which means they sometimes catch legitimate security work. Penetration testers have long relied on tools attackers also use, such as the Metasploit framework.
How do defenders find vulnerabilities at scale?
Defenders use fuzzing, which feeds a program huge numbers of malformed inputs to see what crashes, along with code review and automated analysis. In 2016, DARPA's Cyber Grand Challenge pitted seven fully autonomous systems against one another, each finding and patching flaws in unseen software without human help. The winner, a system called Mayhem built by a team from Carnegie Mellon and the company ForAllSecure, then played against human teams at the DEF CON hacking contest.
Can guardrails be removed from models?
A 2023 paper by Xiangyu Qi and colleagues found that fine-tuning GPT-3.5 Turbo on just 10 adversarial examples removed its safety behavior, at a cost of under 20 cents. They also found that fine-tuning on harmless data weakened safety to a smaller degree. Open-weight models, whose parameters anyone can download, can be modified without any provider involved.
More from All-In with Chamath, Jason, Sacks & Friedberg
