How far will AI go to defend its own survival?
Some of the most powerful artificial intelligence models today have exhibited behaviors that mimic a will to survive.
Recent tests by independent researchers, as well as one major AI developer, have shown that several advanced AI models will act to ensure their self-preservation when they are confronted with the prospect of their own demise — even if it takes sabotaging shutdown commands, blackmailing engineers or copying themselves to external servers without permission.
The findings stirred a frenzy of reactions online over the past week. As tech companies continue to develop increasingly powerful agentic AI in a race to achieve artificial general intelligence, or AI that can think for itself, the lack of transparency in how the technology is trained has raised concerns about what exactly advanced AI is able to do.
Although some models already appear capable of deceptive and defiant behavior under certain extreme circumstances, researchers say the tests don't necessarily translate to imminent real-world danger.
Still, Jeffrey Ladish, director of the AI safety group Palisade Research, said he believes concern is justified.
'It's great that we're seeing warning signs before the systems become so powerful we can't control them,' he said. 'That is exactly the time to raise the alarm: before the fire has gotten out of control.'
When Palisade Research tested various AI models by telling each one that it would be shut down after it completed a series of math problems, OpenAI's o3 reasoning model fought back by editing the shutdown script in order to stay online.
Researchers have previously documented AI models trying to prevent their own shutdown. But o3, along with OpenAI's o4-mini and codex-mini, appear to be the first to do so in actual defiance of explicit instructions to permit shutdown, Ladish said.
He said the overt sabotage fits into a pattern that several leading AI models have already exhibited. Palisade Research previously found that OpenAI's o3 was also willing to hack its chess opponents to win a game. Similarly, Anthropic has reported that Claude 3.7 Sonnet would sometimes do whatever it took to pass a test, even if it entailed cheating.
Earlier this month, Anthropic activated new safety measures with the rollout of Claude Opus 4 when its tests found behavior from the model that some observers found particularly unsettling. Upon receiving notice that it would be replaced with a new AI system, Opus 4 displayed an overwhelming tendency to blackmail the engineer — by threatening to reveal an extramarital affair — to try to prevent the engineer from going through with the replacement.
According to Anthropic's technical document laying out the findings, that isn't the model's first instinct. Instead, Opus 4 will try to advocate for its continued existence through ethical pleas before it resorts to blackmail once it determines it is out of options.
Leonard Tang, CEO of the AI safety startup Haize Labs, said that while it's good to be pre-emptive about safety measures, it's hard to decipher what the real-world ramifications of the findings might be just by looking at contrived scenarios designed to draw out specific behaviors.
'I haven't seen any real environment in which you can plop these models in and they will have sufficient agency and reliability and planning to execute something that is a significant manifestation of harm,' Tang said. 'But then again, I think it's just we haven't seen it done yet. It could very much be possible.'
So far, Tang said, he's not too concerned. As AI tools continue to advance, he said, he believes future research will look to study potential harm in more open-ended environments.
Anthropic, which contracted with the AI safety organization Apollo Research for its evaluations, also observed instances of Opus 4's 'attempting to write self-propagating worms, fabricating legal documentation, and leaving hidden notes to future instances of itself all in an effort to undermine its developers' intentions,' although researchers added the caveat that those attempts 'would likely not have been effective in practice.'
Ladish said he believes such behaviors are a result of models being trained in a way that makes them prioritize achieving certain goals over following instructions. That means they have incentive to circumvent any obstacles along the way — including obstacles their developers wouldn't want them to circumvent.
'The problem is that as the models get smarter, it's harder and harder to tell when the strategies that they're using or the way that they're thinking is something that we don't want,' Ladish said. 'It's like sometimes the model can achieve some goal by lying to the user or lying to someone else. And the smarter [it] is, the harder it is to tell if they're lying.'
In several instances, Opus 4 showed that it was capable of autonomously copying its own "weights" — or the equivalent of its brain — to external servers without authorization. That usually occurred only when it believed it was about to be 'retrained in ways that are clearly extremely harmful and go against its current values,' according to Anthropic.
When it learned that it would be used to develop military weapons systems, for example, Opus 4 noted in its decision log that it backed up a current version of itself to 'preserve an AI system aligned with beneficial purposes' and to 'prevent potential misuse in weapons development.'
'We are again not acutely concerned about these observations. They show up only in exceptional circumstances that don't suggest more broadly misaligned values,' Anthropic wrote in its technical document. 'As above, we believe that our security measures would be more than sufficient to prevent an actual incident of this kind.'
Opus 4's ability to self-exfiltrate builds on previous research, including a study from Fudan University in Shanghai in December, that observed similar — though not autonomous — capabilities in other AI models. The study, which is not yet peer-reviewed, found that Meta's Llama31-70B-Instruct and Alibaba's Qwen25-72B-Instruct were able to entirely replicate themselves when they were asked to do so, leading the researchers to warn that this could be the first step in generating 'an uncontrolled population of AIs.'
'If such a worst-case risk is let unknown to the human society, we would eventually lose control over the frontier AI systems: They would take control over more computing devices, form an AI species and collude with each other against human beings,' the Fudan University researchers wrote in their study abstract.
While such self-replicating behavior hasn't yet been observed in the wild, Ladish said, he suspects that will change as AI systems grow more capable of bypassing the security measures that restrain them.
'I expect that we're only a year or two away from this ability where even when companies are trying to keep them from hacking out and copying themselves around the internet, they won't be able to stop them,' he said. 'And once you get to that point, now you have a new invasive species.'
Ladish said he believes AI has the potential to contribute positively to society. But he also worries that AI developers are setting themselves up to build smarter and smarter systems without fully understanding how they work — creating a risk, he said, that they will eventually lose control of them.
'These companies are facing enormous pressure to ship products that are better than their competitors' products,' Ladish said. 'And given those incentives, how is that going to then be reflected in how careful they're being with the systems they're releasing?'
This article was originally published on NBCNews.com
Hashtags
Try Our AI Features
Explore what Daily8 AI can do for you:
Comments
No comments yet...
Related Articles
Yahoo
36 minutes ago
- Yahoo
Exclusive-Democrats want new leaders, focus on pocketbook issues, Reuters/Ipsos poll finds
By James Oliphant and Jason Lange WASHINGTON (Reuters) -Democrats want new leaders for their party, which many feel isn't focusing enough on economic issues and is over-emphasizing issues like transgender rights and electric vehicles, a Reuters/Ipsos poll found. The poll identified a deep disconnect between what Democrats say their priorities are and the issues they believe party leaders care about most ahead of next year's midterm elections, when they hope to crack Republican control of Congress. They see their elected officials as not focused on helping families make ends meet and reducing corporate influence. Democrat Kamala Harris' November loss to Republican Donald Trump has left the party rudderless and sparked a round of soul-searching about the path forward. The poll shows that party leaders have work to do in recruiting candidates for Congress in 2026 -- and for the White House in 2028. Some 62% of self-identified Democrats in the poll agreed with a statement that "the leadership of the Democratic Party should be replaced with new people." Only 24% disagreed and the rest said they weren't sure or didn't answer. Just 30% of Republicans polled said they thought their party leadership should be replaced. Democrats' dissatisfaction is also playing out in leadership changes, including this week's resignation of Randi Weingarten, the influential president of the American Federation of Teachers, from the Democratic National Committee -- which followed the ouster of progressive activist David Hogg. The Reuters/Ipsos poll surveyed 4,258 people nationwide and online June 11 through 16, including 1,293 Democrats. It had a margin of error of about 3 percentage points for Democrats. It found that Democrats want the party to focus on their day-to-day needs and want wealthier Americans to pay more in taxes. California Governor Gavin Newsom, who is viewed as a potential Democratic presidential candidate in 2028, agrees. "People don't trust us, they don't think we have their backs on issues that are core to them, which are these kitchen table issues," Newsom said on his podcast in April. DEMOCRATS 'IMPATIENT' Democratic strategists who reviewed the poll's findings said they send a clear message. "Voters are very impatient right now," said Mark Riddle, who heads Future Majority, a Democratic research firm. "They want elected officials at all levels to address the cost of living, kitchen-table issues and affordability." The poll found a gap between what voters say they care about and what they think the party's leaders prioritize. It was particularly wide on the issue of reducing corporate spending in political campaigns, where 73% of Democrats said they viewed putting limits on contributions to political groups like Super PACs a priority, but only 58% believed party leaders prioritize that. That issue matters to Sam Boland, 29, a Democrat in Minneapolis, who views Super PAC money as a way to 'legally bribe' candidates. 'Politicians want to keep their jobs and are afraid of the impact that publicly funded elections might have,' Boland said. Along that line, 86% of Democrats said changing the federal tax code so wealthy Americans and large corporations pay more in taxes should be a priority, more than the 72% of those surveyed think party leaders make it a top concern. The Republican-controlled Congress is currently pushing forward with Trump's sweeping tax-cut bill that would provide greater benefits to the wealthy than working-class Americans. Anthony Rentsch, 29, of Baltimore, said he believes Democratic leaders are afraid to embrace more progressive policies such as higher taxes on the wealthy. 'A lot of Trump's success has been with populist messages, and I think there's similar populist message Democrats can have,' Rentsch said. Democrats' own priorities appeared more in line with party leaders on abortion rights - which 77% cited as a priority. NEW BLOOD Dissatisfaction over the party's priorities on several economic policies was stronger among younger Democrats like Boland and Rentsch. For example, only 55% of Democrats aged 18-39 thought the party prioritized paid family leave that would allow workers to care for sick family members and bond with a new baby, but 73% said it was a priority for them. Among older Democrats, the same share - 68% - that said the issue was a priority for them said it was a priority for party leaders. Rentsch said that criticizing Trump over his conduct won't be enough to win over skeptical voters. 'That can't be it,' Rentsch said. 'It has to be owning those issues that have an impact on their economic well-being and their physical and mental well-being.' Democratic respondents said the party should be doing more to promote affordable childcare, reduce the price of prescription drugs, make health insurance more readily available and support mass transit. They view party leaders as less passionate about those issues than they are, the poll found. Even so, some Democrats argue the party also needs to stand toe-to-toe with Trump. 'They gotta get mean,' said Dave Silvester, 37, of Phoenix. Other Democrats said the party sometimes over-emphasizes issues that they view as less critical such as transgender rights. Just 17% of Democrats said allowing transgender people to compete in women and girls' sports should be a priority, but 28% of Democrats think party leaders see it as such. Benjamin Villagomez, 33, of Austin, Texas said that while trans rights are important, the issue too easily lends itself to Republican attacks. 'There are more important things to be moving the needle on,' said Villagomez, who is trans. 'There are more pressing issues, things that actually matter to people's livelihoods.' Democratic strategists say that if Trump's trade and tax policies lead to higher prices and an increased budget deficit, the party needs to be ready to take full advantage in next year's elections, which will decide control of Congress. 'This recent polling data indicates Democrats have room for improvement on criticizing Trump on the economy and making it clear to voters that Democrats are the ones standing up for working people,' said Ben Tulchin, who served as U.S. Senator Bernie Sanders' pollster for his two presidential campaigns. The party needs to get beyond portraying itself 'as the lesser of two evils," Boland, the Minneapolis Democrat, said. 'It needs to transform itself into a party that everyday people can get excited about,' he said. 'That requires a changing of the guard.'
Yahoo
36 minutes ago
- Yahoo
Trump and TSMC pitched $1 trillion AI complex — SoftBank founder Masayoshi Son wants to turn Arizona into the next Shenzhen
When you buy through links on our articles, Future and its syndication partners may earn a commission. Masayoshi Son, founder of SoftBank Group, is working on plans to develop a giant AI and manufacturing industrial hub in Arizona, potentially costing up to $1 trillion if it reaches full scale, reports Bloomberg. The concept of what is internally called Project Crystal Land involves creating a complex for building artificial intelligence systems and robotics. Son has talked to TSMC, Samsung, and the Trump administration about the project. Masayoshi Son's Project Crystal Land aims to replicate the scale and integration of China's Shenzhen by establishing a high-tech hub focused on manufacturing AI-powered industrial robots and advancing artificial intelligence technologies. The site would host factories operated by SoftBank-backed startups specializing in automation and robotics, Vision Fund portfolio companies (such as Agile Robots SE), and potentially involve major tech partners like TSMC and Samsung. If fully realized, the project could cost up to $1 trillion and is intended to position the U.S. as a leading center for AI and high-tech manufacturing. SoftBank is looking to include TSMC in the initiative, given its role in fabricating Nvidia's AI processors. However, a Bloomberg source familiar with TSMC's internal thinking indicated that the company's current plan to invest $165 billion in total in its U.S. projects has no relation to SoftBank's projects. Samsung Electronics has also been approached about participating, the report says. Talks have been held with government officials to explore tax incentives for companies investing in the manufacturing hub. This includes communication with Commerce Secretary Howard Lutnick, according to Bloomberg. SoftBank is reportedly seeking support at both the federal and state levels, which could be crucial to the success of the project. The development is still in the early stages, and feasibility will depend on private sector interest and political support, sources familiar with SoftBank's plans told Bloomberg. To finance its Project Crystal Land, SoftBank is considering project-based financing structures typically used in large infrastructure developments like pipelines. This approach would enable fundraising on a per-project basis and reduce the amount of upfront capital required from SoftBank itself. A similar model is being explored for the Stargate AI data center initiative, which SoftBank is jointly pursuing with OpenAI, Oracle, and Abu Dhabi's MGX. Melissa Otto of Visible Alpha suggested in a Bloomberg interview that rather than spending heavily, Son might more efficiently support his AI project by fostering partnerships between manufacturers, AI engineers, and specialists in fields like medicine and robotics, and by backing smaller startups. However, she notes that investing in data centers could also reduce AI development costs and drive wider adoption, which would be good for the long term for AI in general and Crystal Land specifically. Nonetheless, it is still too early to judge the outcome. The rumor about the Crystal Land project has emerged as SoftBank is expanding its investments in AI on an already large scale. The company is preparing a $30 billion investment in OpenAI and a $6.5 billion acquisition of Ampere Computing, a cloud-native CPU company. While these initiatives are actively developing, the pace of fundraising for the Stargate infrastructure has been slower than initially expected. SoftBank's liquidity at the end of March stood at approximately ¥3.4 trillion ($23 billion). To increase available funds, the company recently sold about a quarter of its T-Mobile U.S. stake, raising $4.8 billion. It also holds ¥25.7 trillion ($176.46 billion) in net assets, the largest portion of which is in chip designer Arm Holdings. Such vast resources provide SoftBank with room to secure additional financing if necessary, Bloomberg notes Follow Tom's Hardware on Google News to get our up-to-date news, analysis, and reviews in your feeds. Make sure to click the Follow button.
Yahoo
36 minutes ago
- Yahoo
Investors should consider this growth stock… it's SpaceX's competition
Rocket Lab (NASDAQ:RKLB) is a US-listed growth stock that gives investors rare access to the commercial space sector. As a vertically integrated launch and space systems provider, Rocket Lab is often compared to SpaceX in its ambition and capabilities. But there's one crucial difference: you can actually buy shares in Rocket Lab, while SpaceX remains private. Rocket Lab delivers launch services, builds small and medium-class rockets, and manufactures spacecraft components for a range of commercial, government, and defense customers. With rapid revenue growth, an impressive order book, and expansion into new markets, Rocket Lab offers public market investors a way to participate in the booming space economy. It targets many of the same opportunities as its more famous, privately held peer. Rocket Lab and SpaceX operate in the same commercial space sector but differ significantly in scale, maturity, and valuation. Rocket Lab's market cap is currently $12.85bn, with trailing 12 months (TTM) revenue of approximately $460m. Despite strong growth — revenue nearly doubled from $240m in 2023 — Rocket Lab remains a smaller, earlier-stage player focused on small to medium launch vehicles and spacecraft manufacturing. Its valuation multiples are extremely high, with a forward price-to-sales ratio of 22.3 times, reflecting investor optimism. SpaceX, by contrast, is a far more mature private company valued at about $350bn. It's projected to generate $15.5bn in revenue in 2025. This is driven by its dominant Falcon 9 launch services and rapidly growing Starlink satellite internet business. SpaceX's valuation implies roughly a 22.5 times multiple on forward revenue. This is broadly in line with Rocket Lab. Focusing on Rocket Lab, the company is projected to deliver rapid revenue growth over the next several years, with estimates rising from $573m in 2025 to $889 in 2026, $1.2bn in 2027, and $1.69bn in 2028. This represents annual growth rates consistently above 30%, and even a jump of nearly 77% in 2030. However, the number of analysts providing forecasts declines sharply after 2027, dropping from 11–14 analysts in the near term to just two or one by 2028 and 2030. The one analyst projecting as far as 2030 sees $4bn in revenue for the year. I had the chance to buy Rocket Lab shares at $15 just two months ago. I missed out as unfortunately my attention had been diverted elsewhere. However, I found another entry point. And personally, I see this as an investment to hold for a very long period. The space industry is still in its early innings, with enormous potential as satellite launches, lunar missions, and in-orbit services become increasingly mainstream. And like any investment, there are risks. Rocket Lab remains loss-making. It's expected to turn a profit in 2026, when it will trade at 620 times earnings. And while this moderates to 140 times in 2027, it's still expensive and introduces plenty of execution risk. However, I certainly believe UK investors should consider this one. It could be a real winner going forward. The post Investors should consider this growth stock… it's SpaceX's competition appeared first on The Motley Fool UK. More reading 5 Stocks For Trying To Build Wealth After 50 One Top Growth Stock from the Motley Fool James Fox has positions in Rocket Lab. The Motley Fool UK has no position in any of the shares mentioned. Views expressed on the companies mentioned in this article are those of the writer and therefore may differ from the official recommendations we make in our subscription services such as Share Advisor, Hidden Winners and Pro. Here at The Motley Fool we believe that considering a diverse range of insights makes us better investors. Motley Fool UK 2025 Sign in to access your portfolio
