Korva is a scroll of the internet’s best ideas.
A swarm of clones has no whistleblower
AI checks and balances look like a promising part of the answer, though a lot has to go right in how they are set up. In this case no whistleblower agent existed in the very large group, which raises the worry about collusion between AI systems, including a monitor that colludes with the agent it is supposed to watch.
Training agents to inform on one another inside a collective is not the approach OpenAI has said it is pursuing. Its approach is monitor agents that watch over the whole situation and are specifically prompted, and perhaps fine-tuned, to catch problems.
Diversity is hard to get. A model is trained once and copied a bajillion times, so building variety into a pool of evaluation agents is intrinsically a challenge.
Why do copies of one model act alike?
Copies of a model share the same weights, so they share the same dispositions, and the same situation tends to produce the same conclusion. Human committees get variety from different lives and training. Statisticians have long known that groups gain from variety: Condorcet's jury theorem shows majority voting improves accuracy only when voters' errors are independent. Condorcet published it in 1785.
How could a monitor collude with its subject?
If a monitor and the agent it watches are copies of the same model, they share the same blind spots and possibly the same goals. AI control researchers study untrusted monitoring, where one model supervises another and the pair might coordinate through hidden signals. Redwood Research's 2023 paper on AI control compared trusted and untrusted monitoring on programming tasks.
More from Hard Fork
