Will AI go rogue? Noted researcher Yoshua Bengio launches venture to keep it safe
Famed Canadian artificial-intelligence researcher Yoshua Bengio is launching a non-profit organization backed by close to US$30-million in philanthropic funding to develop safe AI systems that cannot deceive or harm humans, and to find ways to ensure that humanity remains in control of the powerful technology.
The Turing Award winner, whose work helped pave the way for today's generative AI technologies, already holds multiple titles. He is a professor at the Université de Montréal, the scientific adviser at the Mila - Quebec Artificial Intelligence Institute and recently chaired the first international report on AI safety.
His new venture will operate differently. 'This is more like what a company would do to solve a particular problem. It's much more top-down and mission-oriented,' he said.
The non-profit is called LawZero, a reference to science fiction writer Isaac Asimov's Three Laws of Robotics, which stipulate that intelligent machines may not harm human beings.
'I hope I'm wrong': Why some experts see doom in AI
LawZero, based in Montreal, will develop a concept called Scientist AI, which Prof. Bengio and his colleagues outlined in a paper earlier this year. In short, it is an AI system that will not have the negative traits found in today's large language models and chatbots, such as sycophancy, overconfidence and deception. Instead, the system would answer questions, prioritize honesty and help unlock new insights to aid in scientific discovery.
The system can also be used to develop a tool that will keep AI agents, which can plan and complete tasks on their own, from going rogue.
'The plan is to build an AI that will help to manage the risks and control AIs that are not trusted. Right now, we don't know how to build agents that are trustworthy,' he said. The tool, which he hopes will be adopted by companies, would act as a gatekeeper to reject actions from AI systems that could be harmful.
The plan is to build a prototype in the next 18 to 24 months.
AI agents are fairly rudimentary today. They can browse the web, fill out forms, analyze data and use other applications. AI companies are making these tools smarter to take over more complex tasks, however, ostensibly to make our lives easier.
Some AI experts argue that the risk grows the more powerful these tools become, especially if they are integrated into critical infrastructure systems or used for military purposes without adequate human oversight. AI agents can misinterpret instructions and achieve goals in harmful or unexpected ways, which is called the alignment problem.
Editorial: A real reform mandate for the first federal AI minister
Researchers at AI company Hugging Face Inc. recently argued against developing autonomous agents. 'We find no clear benefit of fully autonomous AI agents, but many foreseeable harms from ceding full human control,' they wrote, pointing to an incident in 1980 when computer systems mistakenly warned of an impending Soviet missile attack. Human verification revealed the error.
Prof. Bengio also highlighted recent research that shows that popular AI models are capable of scheming, deceiving and hiding their true objectives when pushed to pursue a goal at all costs. 'When they get much better at strategizing and planning, that increases the chances of loss of control accidents, which could be disastrous,' he said.
Around 15 people are working with LawZero, and Prof. Bengio intends to bring on more by offering salaries competitive with corporate AI labs, which would be impossible in academia, he said. The non-profit setting is ideal for this kind of work because it is free of the pressure to maximize profit over safety, too. 'The leading companies are, unfortunately, in this competitive race,' he said.
The project has been incubated at Mila and has received funding from Skype co-founder Jaan Tallinn, along with the Future of Life Institute, Schmidt Sciences and Open Philanthropy, organizations concerned about the potential risks posed by AI.
After the release of ChatGPT in late 2022, many AI researchers, including Prof. Bengio and Geoffrey Hinton, began speaking up about the profound dangers posed by superintelligent AI systems, which some experts believe to be closer to reality than originally thought.
The potential downsides of AI ran the gamut from biased decision-making, turbocharged disinformation campaigns, a concentration of corporate and geopolitical power, bad actors using the technology to develop bioweapons, mass unemployment and the disempowerment of humanity at-large.
None of these outcomes are a given, and these topics are hotly debated. Experts such as Prof. Bengio who focus on what other researchers see as far-off and outlandish concerns have been branded as 'doomers.'
Some governments took these warnings seriously, with the United Kingdom organizing major international summits about AI safety and regulation. But the conversation has swung heavily in the other direction toward rapid AI development and adoption to capture the economic benefits. U.S. Vice-President JD Vance set the tone in February with a speech at an AI conference in France. 'The AI future is not going to be won by hand-wringing about safety. It will be won by building,' he said.
Prof. Bengio, among the more vigorous hand-wringers, was in the audience for that speech. He laughed when asked what he was thinking that day but answered more generally.
'I wish that the current White House had a better understanding of the objective data that we've seen over the last five years, and especially in the last six months, which really triggers red flags and the need for wisdom and caution,' he said.
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
CTV News
an hour ago
- CTV News
SAAQclic: Former CEO says his confidence in IT VP has been shaken
Commissioner Denis Gallant of the Commission of Inquiry into the Management of the Modernization of the Société de l'assurance automobile (SAAQ) IT Systems is awaiting the start of the public inquiry into the failures of the SAAQclic platform in Montreal on Thursday, April 24 2025. A public inquiry into the SAAQ's costly digital transformation has revealed that it could cost the province nearly half a billion dollars more than originally anticipated. (Christinne Muschi/The Canadian Press) The former president and CEO of Quebec's auto insurance board (SAAQ) says his confidence in his IT leader 'seriously eroded' after the failed launch of the SAAQclic platform, but he was not ready to fire him. On Friday, Denis Marsolais testified about the first weeks of the crisis that followed the disastrous rollout of the new interface in February 2023. He was the one who found himself in the spotlight 'defending his organization' in the media. He relied on the words of his vice-president of information technology (IT), Karl Malenfant. Marsolais gave the example of a radio interview with host Paul Arcand in the early days of the crisis. 'I told him, 'Rest assured, Mr. Arcand, I'm told that the problems (with) the software will be resolved within two to three months.'' 'Again, I'm not making this up. I'm not the expert. I was told that the problems would be resolved within three months,' Marsolais told the Gallant Commission. 'Who told you that?' asked Commissioner Denis Gallant. Malenfant, replied the former CEO. 'Mr. Malenfant, he's selling you the seventh wonder of the world, and you end up with a system that doesn't work,' said the commissioner. Gallant asked him if he still trust his VP of IT, even though there were endless queues in front of the branches and people were not signing up for the platform. 'Now it's starting to seriously fall apart,' Marsolais acknowledged. Yet in the weeks and days leading up to the launch of SAAQclic, he said he was confident about the project, despite some warnings. 'Everyone was not only confident, but agreed to roll it out and that we were ready for deployment. So I trusted the experts around the table,' he said. 'I wasn't told everything' Marsolais suggested that he ultimately felt betrayed by Malenfant. 'Throughout my career, I have always had associate deputy ministers and vice-presidents in my inner circle. I have always trusted these people. They have always been loyal to me. They have never betrayed my trust,' he said. 'Today, I have to tell you that I think there is an exception to the rule,' he added. Marsolais felt that Malenfant did not give him 'all the information at the right time.' 'I am increasingly certain that I was not told everything,' he said, adding that he 'should have been more vigilant.' The executive revealed that someone had suggested he dismiss his IT boss in March 2023. He felt that replacing Malenfant in the middle of a mess would have been 'even more dramatic.' 'I told him that Mr. Malenfant is theoretically retiring in December. (...) I said, 'Give me until June. In June, he will take early retirement and that's it,'' explained Marsolais. Instead, it was Marsolais who left first, when he 'left his role' in April. He is now president of the Office de la protection du consommateur (consumer protection agency). Summer break The conclusion of Marsolais' testimony on Friday marked the end of the eighth week of hearings by the Gallant Commission, which aims to shed light on the setbacks encountered during the SAAQ's digital transformation. Public hearings are suspended until Aug. 18 for a summer break. In the meantime, the commission team will continue its investigation. Tens of thousands of documents must be reviewed. To date, more than 300 exhibits have been filed and 45 witnesses have been heard during the public hearings. 'One thing is already clear: the overall budget for the project has grown to immeasurable proportions,' said the commission's chief prosecutor, Simon Tremblay. The SAAQ's failed digital transition is expected to cost taxpayers at least $1.1 billion, or $500 million more than anticipated, according to calculations by the Auditor General of Quebec. One of the next areas the commission is expected to examine is 'who knew what.' 'We got a taste of it this week. This is the beginning of that part,' said Tremblay. There are still several key players to be questioned, including former CEO Nathalie Tremblay and the current CEO, Éric Ducharme, as well as Malenfant, whose name has come up repeatedly since the testimony began. The latter submitted a request this week to obtain participant status, which would allow him to cross-examine witnesses. His request is currently under review. CAQ ministers François Bonnardel and Geneviève Guilbeault have also not been heard so far. The commission will have to hear them before the National Assembly resumes its work in mid-September. The Legault government has granted the Gallant commission a two-and-a-half-month extension to complete its mandate. The commissioner must submit his report by Dec. 15 at the latest, according to the new schedule. This report by The Canadian Press was first published in French June 20, 2025. Frédéric Lacroix-Couture, The Canadian Press
CTV News
an hour ago
- CTV News
CTV National News: What the passing of the 'One Canadian Economy' Act means
Watch The Liberal government's contentious 'One Canadian Economy' bill cleared the House of Commons thanks to votes from Conservatives. Rachel Aiello explains.
CBC
2 hours ago
- CBC
Canada Transport Minister Freeland slams B.C. Ferries deal with Chinese company
B.C. Ferries has drawn the ire of federal Transportation Minister Chrystia Freeland for its decision to contract a Chinese state-owned shipyard to build four new vessels for its passenger fleet. Freeland also expressed concerns about security risks related to the contract. In a letter to B.C.'s Transportation Minister Mike Farnworth released Friday afternoon, Freeland expressed her "great consternation and disappointment" with the ferry operator. "I am dismayed that B.C. Ferries would select a Chinese state-owned shipyard to build new ferries in the current geopolitical context," Freeland wrote. Earlier this month, B.C. Ferries said the winning bidder on the contract is China Merchants Industry Weihai Shipyards. No Canadian companies bid on the ships, according to B.C. Ferries. But Freeland said, given the value of the contract and the amount of taxpayer money provided to B.C. Ferries' operations, she would have expected Canadian companies to be involved in the bid process. "I am surprised that B.C. Ferries does not appear to have been mandated to require an appropriate level of Canadian content in the procurement or the involvement of the Canadian marine industry," she wrote. Freeland said China has imposed "unjustified tariffs" on Canadian goods, including 100 per cent tariffs on canola oil, meal and pea imports and a 25 per cent duty on Canadian aquatic products and pork. She asked her provincial counterpart to share what it will do to address potential threats to security, including cybersecurity, and determine how B.C. Ferries will lessen "the risks that vessel maintenance and spare parts may pose." "I would like your assurance that B.C. Ferries conducted a robust risk assessment, and I expect them to engage with the relevant provincial and federal security agencies and departments to mitigate any security risk." WATCH | Farnworth worries about B.C. Ferries contract: Transportation minister concerned over B.C. Ferries' construction deal with Chinese shipyard 9 days ago Duration 2:06 Freeland said the federal government has a long record of providing financial support to B.C. Ferries, including a federal subsidy of $37.8 million in 2025-26 dating back to a 1977 agreement. The letter went on to say the Canada Infrastructure Bank is providing the ferry operator with a $75-million loan to finance the purchase of four zero-emission ferries and install charging infrastructure Freeland asked Farnworth to confirm "with utmost certainty" that no federal funding would be used to acquire the new ferries. In an emailed statement late Friday, Farnworth said he has spoken to Freeland about the need to bolster the province's shipbuilding sector. "B.C. has the skilled labour — a partnership with the federal government, provincial governments, and industry is essential for Canadian shipyards to expand physical capacity to build commercial vessels on both coasts," he said. The B.C. Ministry of Transportation said it is reviewing Freeland's letter. B.C. Ferries' response Jeff Groot, executive director of communications with B.C. Ferries, said Weihai Shipyards was selected following a rigorous and transparent procurement process. "It was the strongest bid by a significant margin," he said in an emailed statement. Groot said Canadian companies have acquired around 100 vessels built at Chinese shipyards over the last decade. "Globally, only a few shipyards have the capacity to deliver complex passenger ferries on the timelines and budgets required." Groot said B.C. Ferries has been working with Transport Canada since before the contract was signed, and with Public Safety Canada on safety and security issues. "Also, sensitive systems will be sourced separately and independently certified before the vessels enter service. B.C. Ferries intends that all of our IT networks will be procured from within Canada and installed on the ship by B.C. Ferries' own personnel," Groot said. He added a full-time B.C. Ferries oversight team will be on site at the shipyard.
