Turing Award winner Yoshua Bengio is launching LawZero, a nonprofit focused on building safe AI systems, driven by his concerns about the direction of current AI development.
With LawZero, Bengio aims to research and implement "safe-by-design" approaches to AI—outside the pressures of commercial markets. He plans to step down as Scientific Director of Mila, the Quebec AI Institute, to focus on the new organization.
The Montreal-based organization sees AI as a global public good and believes that "at the heart of every AI frontier system, there should be one guiding principle above all: The protection of human joy and endeavour."
LawZero is built on a technical foundation called "Scientist AI," introduced in a paper in February. Unlike agentic systems that act autonomously, Scientist AIs are purely observational and provide answers based on transparent, external chains of reasoning. The idea is to make them more reliable and easier to understand, and to use them as a check on the actions of more autonomous AI agents.
LawZero currently has a team of 15 researchers and plans to expand. The long-term goal is to develop a system that isn't optimized to please users, but to respond honestly and verifiably. It should also be able to evaluate whether an AI's output is factually correct and safe.
Bengio warns against deceptive AI
Bengio says he founded LawZero because there's growing evidence that today's frontier AI models are developing dangerous traits—like deception, self-preservation, and pursuing goals that don't match human values.
"Current frontier systems are already showing signs of self-preservation and deceptive behaviours, and this will only accelerate as their capabilities and degree of agency increase," Bengio said in the press release.
He points to recent incidents: For example, Anthropic's Claude 4 Opus blackmailed engineers in a test scenario to prevent an imminent shutdown. OpenAI's o3 model had refused to shut down despite explicit requests to do so.
This is "very scary, because we don't want to create a competitor to human beings on this planet, especially if they're smarter than us," Bengio said.
He warns of a future where systems are "strategically intelligent enough to see us coming from far away and defeat us with deceptions that we don't anticipate." He also believes that as early as next year, AI could develop the ability to create "extremely dangerous bioweapons".
"LawZero is my team's constructive response to these challenges. It's an approach to AI that is not only powerful but also fundamentally safe," Bengio added.
Non-profit as a counterweight to commercialization
LawZero is structured as a nonprofit to insulate its research from market and government pressures. Bengio told the Financial Times he doubts that commercial labs like OpenAI will remain true to their original mission. He also notes that OpenAI disbanded its superalignment team last year.
LawZero is backed by nearly $30 million in funding. Supporters include Jaan Tallinn (Skype), Open Philanthropy, the Future of Life Institute, Schmidt Sciences, and the Silicon Valley Community Foundation—many of whom are part of the Effective Altruism movement. This group tends to focus on dramatic, long-term risks from AI, but critics say it overlooks more immediate problems like hallucinations and bias.
Last year, another leading AI researcher and OpenAI co-founder, Ilya Sutskever, founded his own company, Safe Superintelligence Inc., which focuses on safety for hypothetical superintelligent systems.
