Superintelligent Agents Pose Catastrophic Risks — Can Scientist AI Offer a Safer Path?

The leading AI companies are increasingly focused on building generalist AI agents — systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. Bengio and his colleagues propose the development of a non-agentic AI system that is trustworthy and safe by design, which they call Scientist AI.