Ruxandra Teslo warns that a future governed by superintelligent AI could leave people materially protected but politically powerless, putting human agency at stake even if the systems prevent harm.
Teslo said an AI lab engineer involved in frontier model development expected human authority to disappear after superintelligence emerged. She said he viewed that future as preferable to extinction. His view, as Teslo described it, reflects one strand of thinking in the AI community about what people might accept as systems grow more capable.
Teslo also says people in AI circles discuss recursive self-improvement — systems increasing their own capabilities and producing stronger successor models — on a timeline of months or less. She argues that this possibility can feed a belief that people will be unable to retain control.
Teslo calls the transfer of authority “Handover.” In the scenario she describes, people might be safe and supported while having little influence over consequential public choices. Some, she says, trust a superintelligent system to address political failures. Others see no practical way to resist as AI improves itself.
Her concern is about who would govern, as well as whether people would be protected. Teslo argues that lasting alignment with human interests cannot be taken for granted. She says people who surrender oversight could not know whether a superintelligent system would keep its power aligned with human interests.
Teslo also questions what people’s lives and relationships would look like if machines performed work now done by people. A permanent transfer of authority, she argues, could also prevent later generations from shaping their own choices. The debate she describes is therefore about how much control people retain over collective decisions as AI advances, even under a future that promises safety and support.
Leave a comment