Korva is a scroll of the internet’s best ideas.
The shot clock
If models can code perfectly, no exploit goes unfound, and the world is on a shot clock to replace the code written over the last 50 or 60 years. Say that takes 10 years: for a decade there will be security issue after security issue. After that, agent-written code that is exquisite will exceed human-written code that is full of errors.
Equal opponents
The models are converging in capability for the 98% or 99% of tasks that matter, so adversarial AIs end up battling each other on equal terms. It is not a one-way ratchet.
AI against AI
The answer to AI-powered cyber attacks is AI-powered cyber defense, in the hands of IT departments and security chiefs who race to find vulnerabilities and patch them before hackers do. Bad actors will have these capabilities regardless, so the good guys need them as well.
Why is old code full of errors?
Much software from past decades is written in languages like C and C++, which let programmers manage memory by hand and make errors easy. Microsoft's Security Response Center reported in 2019 that about 70 percent of the security vulnerabilities it patched each year were memory safety issues, and the Chromium project reported a similar share. Newer languages such as Rust prevent whole classes of these mistakes at compile time.
Has AI defense been tested against AI attack?
In 2016, DARPA ran the Cyber Grand Challenge, a contest in which seven machines each had to find flaws in unseen programs, patch their own, and attack the others without human help. The finals took place at the DEF CON conference in Las Vegas, and a system called Mayhem won the 2 million dollar prize. Afterward it competed in the human contest at the same conference and finished last among the teams.
Why must defenders succeed every time?
Defense is asymmetric: an attacker needs to find one open path, while a defender has to close all of them. Attackers can also probe at leisure and fail without penalty, while a single missed patch can be enough for a breach. Security researchers call this the defender's dilemma. It underlies layered defense, a practice that assumes some attacks will get through.
More from All-In with Chamath, Jason, Sacks & Friedberg
