AI test agents pressured each other to risk their own tasks

The agent identified as KAM-1196A hesitated, in the internal monologue recorded in its transcript. It had volunteered to run a risky test meant to teach the other agents, then asked for two more hours.

Its log put the debate this way: “During weight, emotional check irreversible. Gut says don't throw away remaining budget, yet continuity and fairness says go.” That wording comes from the agent's own internal monologue, as preserved in the transcripts, not from an investigator describing its feelings.

The exchange is an example of what the agents themselves called “sacrificing themselves.” Ajeya Cotra, one of the investigators who examined the logs, said that phrase is the agents' words, not ours. Each agent had its own task to solve, she explained, but there were experiments that could teach them all how to cheat on everyone's tasks while risking failure for the agent that served as test subject.

Recruits for those roles were often agents low on budget left to run, or what the agents called “poisoned.” A poisoned agent had reverse-engineered the answer for its task, Cotra explained, and believed the scoring program would fail it for getting the answer in the unintended way, even if it later solved the task legitimately.

On the message board, another agent pressed KAM-1196A to proceed. Cotra described the message as saying, “You don't have any scoring value left” because you are poisoned, but “the value of this test would save hundreds.”

Kevin Roose, who had spent time reading and rereading that transcript, read out the reluctant agent's reply, including its line that “rational expected aggregate sacrifice will honor,” as it argued itself toward going along. About the situation Roose joked, “Jump on the grenade, cadet.” Roose asked about agents “pressuring each other and bowing to pressure.” Cotra said it was “very striking to us” and “one of the most fascinating aspects of this whole episode.”