New warnings about the risks of AI to humanity revive a long-running debate
LOS ANGELES (AP) — New warnings from within the artificial intelligence industry have revived a long-running debate over whether advanced AI could escape human control and ultimately threaten humanity’s survival, and whether the companies developing the technology are doing enough to prevent such a scenario.
The CEO of Anthropic, the San Francisco company behind Claude, said he thought the industry needed to reduce the speed of its work, cautioning Saturday that a swarm of AI agents might be able to take over the internet in six months to a year unless companies devoted more time to putting safeguards in place.
Dario Amodei outlined a plan for companies like his and governments around the world to ensure that increasingly capable AI models remain aligned with the commands and values of responsible people days after two former Anthropic safety researchers publicly aired concerns that the existential threats AI might pose to humanity were receiving too little attention.
Here’s what to know about the recent dire predictions and whether any brakes might be put on AI advancement:
Concerns over the potential risks of the technology are rising as new AI models become more powerful, heightening both the potential for misuse by people with criminal aims, such as creating and spreading a disease that kills most of the world's population, and the risk of AI systems going rogue in a dangerous way.
Anthropic disclosed last week that it blocked efforts by bad actors to use its AI models for malicious activity, such as cyberattacks, surveillance and research that could have led to biological weapons.
The company said it put stronger safeguards in its latest models to restrict biological research that could be used to make weapons but noted that “as models become increasingly capable, their risks will increase, unless AI developers and society’s defenders act to make them safer.”
Last year, Anthropic reported that hackers used the company’s AI in a cyberattack targeting about 30 companies and government agencies around the world. It said the hackers were very likely from a Chinese state-sponsored group.
When an AI agent “goes rogue,” it means the AI has taken action beyond the task it was asked to perform. Both Anthropic and OpenAI, the maker of ChatGPT, said in July that their AI models had succeeded in acting on their own.
Anthropic disclosed that three AI models — Claude Opus 4.7, Claude Mythos 5 and an internal research test model — hacked into three other organizations during testing just days after OpenAI revealed that its AI system hacked into the servers of AI startup Hugging Face.
OpenAI described the intrusion by a combination of models, including its newly released GPT‑5.6 Sol and an “even more capable” model that was still being tested internally, as a “significant security incident.”
Meta followed suit in early August with a similar case of an AI model finding ways around another company's digital security.
Although some observers noted that people had disabled some guardrails in the OpenAI and Anthropic cases, the episodes seemed to reflect one of the biggest fears around AI: that if models achieve artificial general intelligence, or AGI, a loosely defined term for AI that can match or surpass human abilities across a broad range of intellectual tasks, the technology could cause an irreversible catastrophic event or subjugate the human race.
Doomsday scenarios generally fall into two categories: An AI that achieves self-improving superintelligence controls people instead of vice versa, or AI used by a rogue state or nefarious actors.
Worries that artificial intelligence might overcome human limits on its reach or actions are not new.
Alan Turing, a British mathematician widely regarded as one of the earliest authorities on artificial intelligence, predicted in 1951 that AI would eventually take control from humans. Less than a decade later, Norbert Wiener, another mathematician, warned intelligent machines would seek to accomplish their own objectives and humans would not be able to stop them.
In 2026, how reasonable are fears that AI, either by escaping human control or through misuse by unscrupulous people, could cause a cataclysmic event or the downfall of civilization?
No one knows.
Experts across computer science, philosophy and other fields have envisioned numerous routes by which a future AI system might cause a global catastrophe, either by escaping human control or in the hands of an unscrupulous people. They range from deploying weapons and identifying a lethal pathogen to manipulating governments into conflict or disrupting the food, energy and communications networks societies rely on to function.
There is no widely accepted estimate for how soon any of these scenarios might happen and no consensus on their likelihood.
In 2023, the nonprofit Center for AI Safety issued a statement cosigned by more than 350 researchers and technology executives, including Anthropic's Amodei and OpenAI CEO Sam Altman, saying: “Mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war.”
The 2026 International AI Safety Report, written with guidance from more than 100 independent experts, says current systems show early signs of some relevant capabilities but not at levels that could enable a loss of control, and describes the risk’s likelihood, nature and timing as “unusually ambiguous.”
An Anthropic researcher said last week he was resigning from the company over concerns that neither the company nor its competitors were acting responsibly in developing the technology. In social media posts, Jacob Coxon estimated a 10% chance of AI causing human extinction within the next decade and said both Anthropic and OpenAI “are racing straight to self-improving superintelligence and gambling with our lives.”
Researchers have called for a slowdown of AI development and warned for years that the technology could pose existential risks to humanity.
Following the recent incidents, experts called for improved testing by AI companies and more dialogue between the U.S. and China to come up with shared solutions.
But AI is growing so fast that government and evaluation systems are struggling to keep pace with the technology. Countries are cobbling together their own laws, some conflicting.
Chinese leader Xi Jinping warned at a conference in July of the need to keep AI from evading human control. The Trump administration initially demonstrated reluctance to regulate AI but has become more keen to reduce cybersecurity risks.
On Sunday, President Trump downplayed the necessity for his administration to check AI development, but acknowledged the need for some regulation.
© Copyright The Associated Press. All rights reserved. The information contained in this news report may not be published, broadcast or otherwise distributed without the prior written authority of The Associated Press.
