He's the godfather of AI. Now, he has a bold new plan to keep us safe from it.

is a senior reporter for Vox's Future Perfect and co-host of the Future Perfect podcast. She writes primarily about the future of consciousness, tracking advances in artificial intelligence and neuroscience and their staggering ethical implications. Before joining Vox, Sigal was the religion editor at the Atlantic.
The science fiction author Isaac Asimov once came up with a set of laws that we humans should program into our robots. In addition to a first, second, and third law, he also introduced a 'zeroth law,' which is so important that it precedes all the others: 'A robot may not injure a human being or, through inaction, allow a human being to come to harm.'
This month, the computer scientist Yoshua Bengio — known as the 'godfather of AI' because of his pioneering work in the field — launched a new organization called LawZero. As you can probably guess, its core mission is to make sure AI won't harm humanity.
Even though he helped lay the foundation for today's advanced AI, Bengio is increasingly worried about the technology over the past few years. In 2023, he signed an open letter urging AI companies to press pause on state-of-the-art AI development. Both because of AI's present harms (like bias against marginalized groups) and AI's future risks (like engineered bioweapons), there are very strong reasons to think that slowing down would have been a good thing.
But companies are companies. They did not slow down. In fact, they created autonomous AIs known as AI agents, which can view your computer screen, select buttons, and perform tasks — just like you can. Whereas ChatGPT needs to be prompted by a human every step of the way, an agent can accomplish multistep goals with very minimal prompting, similar to a personal assistant. Right now, those goals are simple — create a website, say — and the agents don't work that well yet. But Bengio worries that giving AIs agency is an inherently risky move: Eventually, they could escape human control and go 'rogue.'
So now, Bengio is pivoting to a backup plan. If he can't get companies to stop trying to build AI that matches human smarts (artificial general intelligence, or AGI) or even surpasses human smarts (artificial superintelligence, or ASI), then he wants to build something that will block those AIs from harming humanity. He calls it 'Scientist AI.'
Scientist AI won't be like an AI agent — it'll have no autonomy and no goals of its own. Instead, its main job will be to calculate the probability that some other AI's action would cause harm — and, if the action is too risky, block it. AI companies could overlay Scientist AI onto their models to stop them from doing something dangerous, akin to how we put guardrails along highways to stop cars from veering off course.
I talked to Bengio about why he's so disturbed by today's AI systems, whether he regrets doing the research that led to their creation, and whether he thinks throwing yet more AI at the problem will be enough to solve it. A transcript of our unusually candid conversation, edited for length and clarity, follows.
Sigal Samuel
When people express worry about AI, they often express it as a worry about artificial general intelligence or superintelligence. Do you think that's the wrong thing to be worrying about? Should we only worry about AGI or ASI insofar as it includes agency?
Yoshua Bengio
Yes. You could have a superintelligent AI that doesn't 'want' anything, and it's totally not dangerous because it doesn't have its own goals. It's just like a very smart encyclopedia.
Sigal Samuel
Researchers have been warning for years about the risks of AI systems, especially systems with their own goals and general intelligence. Can you explain what's making the situation increasingly scary to you now?
Yoshua Bengio
In the last six months, we've gotten evidence of AIs that are so misaligned that they would go against our moral instructions. They would plan and do these bad things — lying, cheating, trying to persuade us with deceptions, and — worst of all — trying to escape our control and not wanting to be shut down, and doing anything [to avoid shutdown], including blackmail. These are not an immediate danger because they're all controlled experiments...but we don't know how to really deal with this.
Sigal Samuel
And these bad behaviors increase the more agency the AI system has?
Yoshua Bengio
Yes. The systems we had last year, before we got into reasoning models, were much less prone to this. It's just getting worse and worse. That makes sense because we see that their planning ability is improving exponentially. And [the AIs] need good planning to strategize about things like 'How am I going to convince these people to do what I want?' or 'How do I escape their control?' So if we don't fix these problems quickly, we may end up with, initially, funny accidents, and later, not-funny accidents.
That's motivating what we're trying to do at LawZero. We're trying to think about how we design AI more precisely, so that, by construction, it's not even going to have any incentive or reason to do such things. In fact, it's not going to want anything.
Sigal Samuel
Tell me about how Scientist AI could be used as a guardrail against the bad actions of an AI agent. I'm imagining Scientist AI as the babysitter of the agentic AI, double-checking what it's doing.
Yoshua Bengio
So, in order to do the job of a guardrail, you don't need to be an agent yourself. The only thing you need to do is make a good prediction. And the prediction is this: Is this action that my agent wants to do acceptable, morally speaking? Does it satisfy the safety specifications that humans have provided? Or is it going to harm somebody? And if the answer is yes, with some probability that's not very small, then the guardrail says: No, this is a bad action. And the agent has to [try a different] action.
Sigal Samuel
But even if we build Scientist AI, the domain of 'What is moral or immoral?' is famously contentious. There's just no consensus. So how would Scientist AI learn what to classify as a bad action?
Yoshua Bengio
It's not for any kind of AI to decide what is right or wrong. We should establish that using democracy. Law should be about trying to be clear about what is acceptable or not.
Now, of course, there could be ambiguity in the law. Hence you can get a corporate lawyer who is able to find loopholes in the law. But there's a way around this: Scientist AI is planned so that it will see the ambiguity. It will see that there are different interpretations, say, of a particular rule. And then it can be conservative about the interpretation — as in, if any of the plausible interpretations would judge this action as really bad, then the action is rejected.
Sigal Samuel
I think a problem there would be that almost any moral choice arguably has ambiguity. We've got some of the most contentious moral issues — think about gun control or abortion in the US — where, even democratically, you might get a significant proportion of the population that says they're opposed. How do you propose to deal with that?
Yoshua Bengio
I don't. Except by having the strongest possible honesty and rationality in the answers, which, in my opinion, would already be a big gain compared to the sort of democratic discussions that are happening. One of the features of the Scientist AI, like a good human scientist, is that you can ask: Why are you saying this? And he would come up with — not 'he,' sorry! — it would come up with a justification.
The AI would be involved in the dialogue to try to help us rationalize what are the pros and cons and so on. So I actually think that these sorts of machines could be turned into tools to help democratic debates. It's a little bit more than fact-checking — it's also like reasoning-checking.
Sigal Samuel
This idea of developing Scientist AI stems from your disillusionment with the AI we've been developing so far. And your research was very foundational in laying the groundwork for that kind of AI. On a personal level, do you feel some sense of inner conflict or regret about having done the research that laid that groundwork?
Yoshua Bengio
I should have thought of this 10 years ago. In fact, I could have, because I read some of the early works in AI safety. But I think there are very strong psychological defenses that I had, and that most of the AI researchers have. You want to feel good about your work, and you want to feel like you're the good guy, not doing something that could cause in the future lots of harm and death. So we kind of look the other way.
And for myself, I was thinking: This is so far into the future! Before we get to the science-fiction-sounding things, we're going to have AI that can help us with medicine and climate and education, and it's going to be great. So let's worry about these things when we get there.
But that was before ChatGPT came. When ChatGPT came, I couldn't continue living with this internal lie, because, well, we are getting very close to human-level.
Sigal Samuel
The reason I ask this is because it struck me when reading your plan for Scientist AI that you say it's modeled after the platonic idea of a scientist — a selfless, ideal person who's just trying to understand the world. I thought: Are you in some way trying to build the ideal version of yourself, this 'he' that you mentioned, the ideal scientist? Is it like what you wish you could have been?
Yoshua Bengio
You should do psychotherapy instead of journalism! Yeah, you're pretty close to the mark. In a way, it's an ideal that I have been looking toward for myself. I think that's an ideal that scientists should be looking toward as a model. Because, for the most part in science, we need to step back from our emotions so that we avoid biases and preconceived ideas and ego.
Sigal Samuel
A couple of years ago you were one of the signatories of the letter urging AI companies to pause cutting-edge work. Obviously, the pause did not happen. For me, one of the takeaways from that moment was that we're at a point where this is not predominantly a technological problem. It's political. It's really about power and who gets the power to shape the incentive structure.
We know the incentives in the AI industry are horribly misaligned. There's massive commercial pressure to build cutting-edge AI. To do that, you need a ton of compute so you need billions of dollars, so you're practically forced to get in bed with a Microsoft or an Amazon. How do you propose to avoid that fate?
Yoshua Bengio
That's why we're doing this as a nonprofit. We want to avoid the market pressure that would force us into the capability race and, instead, focus on the scientific aspects of safety.
I think we could do a lot of good without having to train frontier models ourselves. If we come up with a methodology for training AI that is convincingly safer, at least on some aspects like loss of control, and we hand it over almost for free to companies that are building AI — well, no one in these companies actually wants to see a rogue AI. It's just that they don't have the incentive to do the work! So I think just knowing how to fix the problem would reduce the risks considerably.
I also think that governments will hopefully take these questions more and more seriously. I know right now it doesn't look like it, but when we start seeing more evidence of the kind we've seen in the last six months, but stronger and more scary, public opinion might push sufficiently that we'll see regulation or some way to incentivize companies to behave better. It might even happen just for market reasons — like, [AI companies] could be sued. So, at some point, they might reason that they should be willing to pay some money to reduce the risks of accidents.
Sigal Samuel
I was happy to see that LawZero isn't only talking about reducing the risks of accidents but is also talking about 'protecting human joy and endeavor.' A lot of people fear that if AI gets better than them at things, well, what is the meaning of their life? How would you advise people to think about the meaning of their human life if we enter an era where machines have both agency and extreme intelligence?
Yoshua Bengio
I understand it would be easy to be discouraged and to feel powerless. But the decisions that human beings are going to make in the coming years as AI becomes more powerful — these decisions are incredibly consequential. So there's a sense in which it's hard to get more meaning than that! If you want to do something about it, be part of the thinking, be part of the democratic debate.
I would advise us all to remind ourselves that we have agency. And we have an amazing task in front of us: to shape the future.

Try Our AI Features

Explore what Daily8 AI can do for you:

Learn with AIFact CheckingText to SpeechAI Summary

Related Articles

Entrepreneur

26 minutes ago

• Entrepreneur

This Windows 11 Pro Upgrade Is a No-Brainer at $15

Disclosure: Our goal is to feature products and services that we think you'll find interesting and useful. If you purchase them, Entrepreneur may get a small share of the revenue from the sale from our commerce partners. You don't need to overhaul your company's hardware to boost performance. Sometimes, the smartest investment is in the operating system itself. Right now, business leaders can grab a lifetime license to Microsoft Windows 11 Pro for just $14.97 (regularly $199) through July 20—a powerful upgrade for any professional environment. Whether you're running a solo consultancy, scaling a startup, or managing a growing remote team, Windows 11 Pro offers the security, productivity, and performance enhancements your operation demands. It's designed for power users and professionals who can't afford downtime, slow systems, or limited features. With tools like BitLocker encryption, Hyper-V virtualization, Azure AD support, and Windows Sandbox, this version goes far beyond the home edition. For entrepreneurs juggling sensitive data or developers working in isolated environments, these are necessities. The modernized interface and snap layouts make multitasking a breeze, while Windows Copilot, the built-in AI assistant, helps you summarize content, generate code, or change settings in seconds. It's a productivity win, especially when paired with Teams and voice-to-text capabilities. For small business owners navigating hybrid teams or IT managers juggling multiple devices, Windows 11 Pro also simplifies device management. With features like Group Policy support and remote desktop functionality, you can easily configure, monitor, and secure multiple machines from a single point of control. This is especially useful for businesses with distributed teams or those handling sensitive client data. Plus, compatibility with Microsoft Intune and third-party endpoint management tools means you can streamline onboarding and enforce security policies—without having to invest in expensive IT infrastructure. This is a lifetime license, so you only pay once, and never worry about renewals or subscriptions again. Get Windows 11 Pro for just $14.97 (reg. $199) through July 20. Microsoft Windows 11 Pro See Deal StackSocial prices subject to change.

CNN

30 minutes ago

• CNN

How your AI prompts could harm the environment

AI Sustainability Climate change EconomyFacebookTweetLink Follow Sign up for CNN's Life, But Greener newsletter. Our limited newsletter series guides you on how to minimize your personal role in the climate crisis — and reduce your eco-anxiety. Whether it's answering work emails or drafting wedding vows, generative artificial intelligence tools have become a trusty copilot in many people's lives. But a growing body of research shows that for every problem AI solves, hidden environmental costs are racking up. Each word in an AI prompt is broken down into clusters of numbers called 'token IDs' and sent to massive data centers — some larger than football fields — powered by coal or natural gas plants. There, stacks of large computers generate responses through dozens of rapid calculations. The whole process can take up to 10 times more energy to complete than a regular Google search, according to a frequently cited estimation by the Electric Power Research Institute. So, for each prompt you give AI, what's the damage? To find out, researchers in Germany tested 14 large language model (LLM) AI systems by asking them both free-response and multiple-choice questions. Complex questions produced up to six times more carbon dioxide emissions than questions with concise answers. In addition, 'smarter' LLMs with more reasoning abilities produced up to 50 times more carbon emissions than simpler systems to answer the same question, the study reported. 'This shows us the tradeoff between energy consumption and the accuracy of model performance,' said Maximilian Dauner, a doctoral student at Hochschule München University of Applied Sciences and first author of the Frontiers in Communication study published Wednesday. Typically, these smarter, more energy intensive LLMs have tens of billions more parameters — the biases used for processing token IDs — than smaller, more concise models. 'You can think of it like a neural network in the brain. The more neuron connections, the more thinking you can do to answer a question,' Dauner said. Complex questions require more energy in part because of the lengthy explanations many AI models are trained to provide, Dauner said. If you ask an AI chatbot to solve an algebra question for you, it may take you through the steps it took to find the answer, he said. 'AI expends a lot of energy being polite, especially if the user is polite, saying 'please' and 'thank you,'' Dauner explained. 'But this just makes their responses even longer, expending more energy to generate each word.' For this reason, Dauner suggests users be more straightforward when communicating with AI models. Specify the length of the answer you want and limit it to one or two sentences, or say you don't need an explanation at all. Most important, Dauner's study highlights that not all AI models are created equally, said Sasha Luccioni, the climate lead at AI company Hugging Face, in an email. Users looking to reduce their carbon footprint can be more intentional about which model they chose for which task. 'Task-specific models are often much smaller and more efficient, and just as good at any context-specific task,' Luccioni explained. If you are a software engineer who solves complex coding problems every day, an AI model suited for coding may be necessary. But for the average high school student who wants help with homework, relying on powerful AI tools is like using a nuclear-powered digital calculator. Even within the same AI company, different model offerings can vary in their reasoning power, so research what capabilities best suit your needs, Dauner said. When possible, Luccioni recommends going back to basic sources — online encyclopedias and phone calculators — to accomplish simple tasks. Putting a number on the environmental impact of AI has proved challenging. The study noted that energy consumption can vary based on the user's proximity to local energy grids and the hardware used to run AI partly why the researchers chose to represent carbon emissions within a range, Dauner said. Furthermore, many AI companies don't share information about their energy consumption — or details like server size or optimization techniques that could help researchers estimate energy consumption, said Shaolei Ren, an associate professor of electrical and computer engineering at the University of California, Riverside who studies AI's water consumption. 'You can't really say AI consumes this much energy or water on average — that's just not meaningful. We need to look at each individual model and then (examine what it uses) for each task,' Ren said. One way AI companies could be more transparent is by disclosing the amount of carbon emissions associated with each prompt, Dauner suggested. 'Generally, if people were more informed about the average (environmental) cost of generating a response, people would maybe start thinking, 'Is it really necessary to turn myself into an action figure just because I'm bored?' Or 'do I have to tell ChatGPT jokes because I have nothing to do?'' Dauner said. Additionally, as more companies push to add generative AI tools to their systems, people may not have much choice how or when they use the technology, Luccioni said. 'We don't need generative AI in web search. Nobody asked for AI chatbots in (messaging apps) or on social media,' Luccioni said. 'This race to stuff them into every single existing technology is truly infuriating, since it comes with real consequences to our planet.' With less available information about AI's resource usage, consumers have less choice, Ren said, adding that regulatory pressures for more transparency are unlikely to the United States anytime soon. Instead, the best hope for more energy-efficient AI may lie in the cost efficacy of using less energy. 'Overall, I'm still positive about (the future). There are many software engineers working hard to improve resource efficiency,' Ren said. 'Other industries consume a lot of energy too, but it's not a reason to suggest AI's environmental impact is not a problem. We should definitely pay attention.'

Gizmodo

an hour ago

• Gizmodo

The $14 Billion AI Google Killer

A new AI darling is making waves in Silicon Valley. It's called Perplexity, and according to reports, both Meta and Apple have quietly explored acquiring it. Valued at a staggering $14 billion following a May funding round, the startup is being hailed as a revolutionary threat to Google Search's search dominance. But here's the thing: it mostly just summarizes web results and sends you links. So why the frenzy? Perplexity billed itself an 'answer engine.' You ask a question, and it uses large language models to spit out a human-sounding summary, complete with footnotes. It's essentially ChatGPT with a bibliography. You might ask for the best books about the French Revolution or a breakdown of the Genius Act. In seconds, it generates a paragraph with links to Wikipedia, news outlets, or Reddit threads. Its pitch is a cleaner, ad-free, chatbot-driven search experience. No SEO junk, no scrolling. But critics say it's little more than a glorified wrapper around Google and OpenAI's APIs, with minimal proprietary tech and lots of smoke. It's fast, clean, and slick. But, they argue, at its core, it's mostly just reorganizing the internet. Big Tech's Obsession That hasn't stopped the hype. In May 2025, the San Francisco, California based company closed another $500 million funding round, pushing its valuation to $14 billion, a sharp increase from its $9 billion valuation in December 2024. Jeff Bezos, via the Jeff Bezos Family Fund, and Nvidia are among its notable backers And now, tech giants are circling. According to Bloomberg, Apple has held talks about acquiring Perplexity. Meta has also reportedly considered the move, though no formal offers have been confirmed. The logic is clear. Perplexity is fast-growing and increasingly seen as a 'Google killer,' especially among tech influencers and X power users. Traffic to its site has exploded in recent months. The company now offers a Chrome extension, mobile app, and a Pro version that gives users access to top-tier AI models like GPT-4 and Claude. Still, it's unclear what exactly makes Perplexity worth $14 billion, other than the fact that it's riding the AI wave. Why AI Skeptics Are Rolling Their Eyes For AI skeptics, Perplexity's rise is yet another example of hype outpacing substance. The site doesn't train its own models. It's not building new infrastructure. It's not revolutionizing search. It's just offering a polished interface to ask questions and get AI-generated summaries pulled from public websites. There are also growing concerns about how Perplexity sources its information. A number of news organizations, including The New York Times, Forbes, and Wired, have accused the company of plagiarizing and scraping content without permission or proper attribution. Journalists and publishers warn that this kind of AI-powered search experience threatens to cannibalize news traffic while giving little back to content creators. On June 20, the BBC became the latest outlet to threaten legal action against Perplexity AI, alleging that the company is using BBC content to train its 'default AI model,' according to the Financial Times. Perplexity CEO Aravind Srinivas has defended the company as an 'aggregator of information.' In July 2024, the startup launched a revenue-sharing program to address the backlash. 'We have always believed that we can build a system where the whole Internet wins,' Srinivas said at the time. So Why the Gold Rush? Simple. Search is money. Google earned $50.7 billion from search ads in the first quarter, a 9.8% increase year over year. If Perplexity can convince even a small share of users to switch, and then monetize that experience, it becomes a real threat. Apple and Meta, both increasingly wary of relying on Google, see Perplexity as a fast track into the AI search race. But the stakes go even deeper. Whoever controls the next search interface controls the user. Just as Google replaced Yahoo, Perplexity could theoretically replace Google. That's why Big Tech wants in, even if it's not entirely clear what they're buying.