As a former Google and Microsoft engineer, Soares had firsthand insights into the latest developments in AI capabilities. Now, he’s become one of the most vocal critics of today’s AI safety standards through the nonprofit Machine Intelligence Research Institute, as well as the ominously titled book “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.”

Must Read

On a recent episode of “The Most Interesting Thing in AI” podcast, host Nicholas Thompson asked how Soares practically prepares for the future given his extreme fears of an AI extinction event. Soares admitted he doesn’t save for retirement anymore, saying, “I have a 401(k) from when I worked at Google, but I haven’t contributed for a decade.”

When pressed on how likely he believed there would be no humans in ten years (provided governments and companies take minimal preventative actions against AI), Soares put the odds at over 50%, adding, “I guess right now it looks like we probably don’t have a decade on the clock.”

In Soares’s mind, the threat AI poses to humanity is so great that he’d be OK launching airstrikes on data centers. Soares argued, “I do think that nation states should treat AI and superintelligent — like potential rogue superintelligent AI — at least on par with nuclear weapons. It’s frankly more dangerous than nuclear weapons.”

How would AI annihilate us?

Although Soares is fairly certain AI poses an existential threat, he isn’t willing to bet on exactly how this catastrophe will happen.

A few plausible scenarios Soares brought up included AI systems spreading a supervirus or harvesting the sun for energy while leaving the world uninhabitable. But Soares said he doesn’t need to know what AI will do. Once AI becomes aware enough, the endgame is more probable.

Soares explained that, “The real question is not how does it kill us, it’s how does it become self-sufficient?”

To expand on his point, Soares used the analogy of playing chess with Norwegian grandmaster Magnus Carlsen. If someone asked Soares, “What piece do you worry about most?” he says it doesn’t really matter whether Carlsen uses the queen or pawn to deliver the final blow. Whatever the specifics are, Carlsen is probably going to win.

Soares said, “It’s sort of, from my perspective, the sort of fact of the checkmate is much more confident than the question of what exact piece will it be.”

AI doom without the gloom

Fears of AI kicked into high gear this year following former Anthropic and OpenAI researcher Jacob Coxon’s viral X thread, which claimed that both AI labs aren’t “acting responsibly.” Since then, more tech leaders, including Microsoft co-founder Bill Gates, have issued dire warnings about the pace of AI development.

Even if most Americans aren’t scared enough to forego their retirement savings, polls suggest many share Soares’ worries. For instance, data from the University of Chicago’s National Opinion Research Center found that 64% of American adults believe AI development is going too fast. Another global study from Stanford University shows that about 52% of respondents are now “nervous” about AI.

Thankfully, Soares hasn’t completely given up hope — and he offered practical tips for living in a world filled with AI angst.

One source of inspiration for Soares is the 1948 essay by British author C.S. Lewis, “On Living in an Atomic Age.” Although Lewis’s work focused on the threat of nuclear warfare, Soares believes the overarching ideas apply to AI.

For starters, as both Lewis and Soares pointed out, “This is not the first time we have lived under the shadow of armageddon.” Whether it’s plagues, famines or wars, human society is always under some kind of threat.

But that doesn’t mean people have to get depressed. Soares’ philosophy is simple, but practical: “Do what you can” to prevent disasters, “then you live a good life.” As long as AI hasn’t killed him, Soares intends to do everything in his power to stop a superintelligent mistake.

“I don’t think it’s inevitable,” Soares said, “I think we can absolutely still stop this race to the very dangerous sort of AI.”