KorvaJoin the waitlist
Tech

A Long Pause on Superintelligence Could Cost More Lives Than It Saves

When safety improves very fast a short pause is enough, when it improves slowly waiting costs more lives than it saves, and only in between does a delay of months or a few years pay off.

AI image
The a16z Show · How Fast Should We Build Superintelligence? | Nick Bostrom

About 150,000 people die each day, largely from aging and disease. In the timing calculation of Nick Bostrom, the Oxford philosopher who wrote the book Superintelligence, those continuing deaths are the cost counted against delaying superintelligence, artificial intelligence far smarter than people.

He asks a narrow question: if the goal were to help people alive now live as long as possible, how quickly should development proceed?

The question answers the warning from Eliezer Yudkowsky and Nate Soares, the authors of the 2025 book "If anyone builds it, everyone dies." Bostrom counters with his own line, "if nobody builds it, everyone dies." He notes that 95% of everyone born before now has already died, and those still alive are on track to follow within decades without major biomedical breakthroughs.

Safe superintelligence, in his account, could provide those breakthroughs by slowing or stopping aging. Without aging, he estimates, human life expectancy would be roughly 1,200 to 1,400 years, a figure he calls conservative because safer conditions could also reduce deaths from accidents and other causes.

The tradeoff, then, is between two kinds of lives saved. Extra time might allow more safety research and lower the risk of a dangerous AI transition. But every year of delay also means millions more deaths from age and disease.

That balance turns on how fast safety improves during a pause. If risk falls very quickly, a wait of months can be enough to make the system much safer. If risk falls very slowly, waiting saves fewer expected lives than are lost each year to natural death, so waiting does not pay.

Only in the middle case, where safety improves at an intermediate pace, does the model favor some delay. Across a wide range of assumptions, Bostrom finds, the best delay is months or a few years, not decades.

He also argues that the earliest period after a nearly final system exists could be unusually productive. Researchers could study it, test it, and deploy it gradually under close watch. After a few years, he expects those additional safety gains to taper off.

Questions people ask

Why would life expectancy be 1,200 to 1,400 years if aging stopped?

If aging stopped, people would still die from accidents, infections, and other causes, but those strike far fewer people each year than aging does. Without aging, life expectancy would be roughly 1,200 to 1,400 years, and Bostrom calls this conservative because a safer world could reduce those other dangers too.

Why would the first few years of a pause help safety more than later years?

At first, researchers would have a concrete, nearly complete system to test rather than theories about future systems. They could run experiments, watch for failures, and fix problems before wide deployment. Once the obvious lessons are learned, further work tends to bring smaller improvements.

Does this argument settle how fast we should build superintelligence?

No. Bostrom describes longer life as one part of the decision, alongside existential risk, fairness to future generations, and other values. A full decision would weigh all of those, and different values could point toward more or less caution.

More in Tech