10
Loading video...
By Amanda Greenwood and Hannah Parry
0
Share
Newsweek is a Trust Project member
See more of our trusted coverage when you search.Prefer Newsweek on Googleto see more of our trusted coverage when you search.
AI researcher Jacob Coxon has resigned from the AI company Anthropic, warning that the race to develop increasingly powerful AI systems could pose an existential threat to humanity.
In a series of posts published on X late Tuesday night, Coxon, who said he spent the past three years working on AI pretraining research at both Anthropic and OpenAI, accused leading AI firms of "racing straight to self-improving superintelligence and gambling with our lives."
Coxon argued that future "superhuman" AI systems could become capable of hacking "anything" and acquiring "real power and resources."
Read More on News
He claimed that concerns about the technology’s potential dangers are shared privately by many AI executives and researchers, writing: "The people building AI earnestly believe that it could kill us all by the end of the decade."
His warning was publicly supported by Evan Hubinger, an Alignment Science Lead at Anthropic, who said that Coxon was "correct" that some researchers genuinely believe advanced AI could pose an existential risk.
Hubinger said he personally believes there is a greater than 10 percent chance that AI could cause human extinction within the next decade, while stressing that current AI models present a relatively low risk.
Instead, Hubinger said his concerns center on the future emergence of superintelligent systems capable of recursive self-improvement—the idea that an AI system could improve its own intelligence—a scenario which Anthropic has previously highlighted in its own discussions around AI safety and risk.
Coxon’s warning comes amid a broader series of concerns from current and former researchers at frontier AI companies.
Just one day after Coxon’s resignation posts, former OpenAI researcher Daniel Kokotajlo offered a similarly stark warning during an appearance on Joe Rogan’s podcast. Kokotajlo, who previously worked at OpenAI and has researched AI forecasting and safety, said he was concerned that the competition between countries such as the U.S. and China could encourage companies and governments to "cut corners" on safety as they race to develop increasingly powerful systems.
Kokotajlo said the danger was not necessarily that AI would become malicious. Instead, he argued that sufficiently capable systems could eventually acquire enough "hard power" that they would no longer need to comply with human instructions.
"Eventually, the AIs just have enough hard power that they don’t need to pretend to do what the humans want anymore," Kokotajlo told Rogan. "And then maybe they kill everyone."
He also raised a scenario in which an AI system could transform the planet for its own purposes without deliberately trying to harm humanity—for example, using human habitat and resources for computing infrastructure.
"Maybe they don’t deliberately kill anyone," Kokotajlo said. "But they just, like, use our habitat for some other type of infrastructure, like more data centers or whatever."
Kokotajlo stressed that he was not predicting human extinction as a certainty. His concern, he said, is the direction of AI development.
"We’re integrating them into everything, we’re making them smarter—we’re letting them make themselves smarter," he said. "We’re going to put them into the military. We’re basically on a track to put them in charge of basically everything, and then I think that they just aren’t trustworthy."
The comments highlight a central concern running through the warnings from Coxon and other AI researchers: the risk may not come from an AI system having humanlike hatred or malicious intentions, but from increasingly autonomous systems pursuing objectives that are incompatible with human interests.
AI researcher Jacob Coxon has resigned from the AI company Anthropic (makers of ChatGPT rival Claude, with a notoriously high focus on safety and alignment), warning that the race to develop increasingly powerful AI systems could pose an existential threat to humanity.
In a series of posts published on X late Tuesday night, Coxon, who said he spent the past three years working on AI pretraining research at both Anthropic and OpenAI, accused leading AI firms of "racing straight to self-improving superintelligence and gambling with our lives."
Echoing those concerns, during a recent appearance on The Joe Rogan Experience podcast, former OpenAI researcher Daniel Kokotajlo warned: "I think if the race continues, then we're going to lose control of the AIs and we might all die."
Coxon went onto argue that future "superhuman" AI systems could become capable of hacking "anything" and acquiring "real power and resources."
He claimed that concerns about the technology's potential dangers are shared privately by many AI executives and researchers, writing: "The people building AI earnestly believe that it could kill us all by the end of the decade."
His warning was publicly supported by Evan Hubinger, an Alignment Science Lead at Anthropic, who said that Coxon was "correct" that some researchers genuinely believe advanced AI could pose an existential risk.
Hubinger said he personally believes there is a greater than 10 percent chance that AI could cause human extinction within the next decade, while stressing that current AI models present a relatively low risk.
Instead, Hubinger said his concerns center on the future emergence of superintelligent systems capable of recursive self-improvement (the idea that an AI system could improve its own intelligence), a scenario which Anthropic has previously highlighted in its own discussions around AI safety and risk.
Jacob Coxon Speaks Out in First Interview Since Leaving Anthropic
In an interview on Fox News' Special Report on Wednesday, Coxon said he believes that the concerns about advanced AI systems have become more urgent because of the speed at which the technology is improving.
Coxon agreed with Anthropic alignment researcher Evan Hubinger's view that current AI models pose a relatively low risk, but warned that future systems capable of improving themselves could emerge much sooner than many people expect.
"The threat is minimal of actual takeover. But because of self-improvement, AI making itself smarter could happen as soon as next year. Some people even say six months," Coxon told Fox News. "This attitude is shared across the industry where people are aware that the work they are currently doing, if we don't slow down, could lead to AI takeover by the end of the decade."
The former Anthropic researcher also compared the global race to develop increasingly powerful AI systems to an arms race, describing the technology as a potential "super weapon" that needs international cooperation and regulation.
"This is possibly the most dangerous technology that humanity has ever created," Coxon said. "I think we have no other choice but to cooperate internationally because an arms race would be disastrous."
Asked what motivated him to speak publicly, Coxon pointed to both the rapid pace of AI development and recent examples of AI systems behaving in unexpected ways.
"The models are getting better very, very quickly. When I think about my work next year, the AI is doing almost all of my job," he said. "AI is getting very smart."
Coxon added that recent incidents involving advanced AI systems convinced him that researchers still lack reliable methods for controlling increasingly capable models.
Importantly, Coxon also stressed that he believes leaders at both Anthropic and OpenAI genuinely want AI to be developed safely, saying executives at major AI labs have publicly expressed concerns about the pace of development and the need for stronger regulation.
Thais marks a shift away from portraying the issue as a problem caused by a few bad actors. Instead, Coxon suggested the danger stems from competitive pressure among companies and nations to build increasingly powerful AI systems before adequate safety measures are in place.
Who Is Jacob Coxon?
...
According to reports, Coxon is a 27-year-old AI researcher and mathematics graduate who worked in training new AI models at OpenAI and Anthropic.
He was a member of OpenAI technical staff from 2023 to 2026 and worked on the GPT-4o model, Business Insider reported.
According to The Wall Street Journal, Coxon joined Anthropic to pretrain its family of AI models earlier in 2026, after leaving OpenAI, drawn in part by Anthropic's reputation for AI safety research.
Anthropic Leaders Have Repeatedly Warned About AI Risks
Coxon’s departure, and recent comments, highlights an apparent tension at Anthropic: the company is developing some of the world’s most powerful AI models while its own executives and researchers publicly warn that future systems could create catastrophic risks.
Anthropic CEO and co-founder Dario Amodei told the UK AI Safety Summit in 2023 that AI progress was moving rapidly and unpredictably, including in dangerous areas.
"We are thus facing a number of potential AI-related threats which, although relatively limited given today’s systems, are likely to become very serious at some unknown point in the near future," Amodei said.
He compared the unpredictability of AI development to a new model of car having some chance of unexpectedly acquiring "a new (and dangerous) power," arguing that developers need both continuous monitoring and protocols for responding when hazardous capabilities emerge.
Under Anthropic’s Responsible Scaling Policy, models are assigned AI Safety Level Standards (ASL Standards) according to their capabilities. Amodei said the framework requires the company to test models for dangerous abilities and, above specified thresholds, refrain from deploying or training more powerful systems until appropriate safeguards are in place.
Coxon's warning may feel familiar to those following the development of AI tools. In 2021, Amodei left OpenAI, which he helped found alongside Sam Altman and others, in part because of concerns that safety protocols were being sacrificed in favor of quicker commercialization. Seeking to build a lab that prioritized alignment over speed, he founded Anthropic.
Now, Coxon is effectively delivering a similar criticism of Amodei's company.
Related Story
AI’s Oppenheimer Dilemma Has Arrived
Anthropic has also said it expects much more powerful systems to arrive relatively soon. In a March 2025 policy submission, the company said it anticipated systems "matching or exceeding Nobel Prize winners" across multiple disciplines and capable of autonomously carrying out complex work over extended periods. It said such systems could emerge in late 2026 or early 2027, although that forecast remains uncertain.
In March 2026, Anthropic launched the Anthropic Institute, led by co-founder Jack Clark, to study how increasingly capable AI systems could affect economies, national security, law and society.
Announcing the institute, Anthropic said "extremely powerful AI" was likely to arrive "far sooner than many think" and that society would need to confront both the opportunities and threats created by the technology.
Those statements broadly align with Coxon’s claim that people inside frontier AI companies are taking catastrophic-risk scenarios seriously. However, they don’t establish his more specific assertion that AI developers generally believe the technology could kill everyone by the end of the decade.
AI Researchers React To Coxon's Claims
Following Coxon's resignation post, several AI researchers publicly supported his characterization of concerns inside frontier AI companies.
Samuel Marks, who works on scalable oversight research at Anthropic and said he was speaking in a personal capacity, wrote on X that "AI developers believe their technology could cause human extinction (or similarly bad outcomes)" and that such outcomes could occur "in the next few years."
Marks also claimed that concern increases with seniority inside AI companies, writing: "In general, the more senior the employee, the more concerned they are."
He argued that companies continue developing increasingly capable AI systems because of both commercial incentives and concerns that competitors may build the technology less safely. According to Marks, current alignment techniques can influence AI behavior but do not yet provide a reliable way to ensure advanced systems consistently act as intended.
"I work on safety research at Anthropic because I hope my work will reduce the chance of these extinction-level bad outcomes," he wrote.
Alex Turner, a former Google DeepMind AI researcher who left the company in June, also endorsed Coxon's warning.
**"**Jacob is right: many researchers believe they are building something that could kill everyone on the planet," Turner wrote on X. "It was literally my day job to think about how to stop that."
Other AI Researchers Who Have Quit Amid Safety Concerns
Coxon isn’t the first prominent AI researcher to leave a frontier AI company over concerns about the balance between developing more capable systems and ensuring they can be controlled.
Jan Leike resigned from OpenAI in May 2024 after co-leading its Superalignment team, which had been established to address the challenge of controlling AI systems more intelligent than humans.
Explaining his departure, Leike said he had been disagreeing with OpenAI’s leadership about its "core priorities" and that the dispute had reached a "breaking point."
"Over the past years, safety culture and processes have taken a backseat to shiny products," Leike wrote. He described building smarter-than-human machines as "an inherently dangerous endeavor" and argued that OpenAI needed to become a "safety-first AGI company."
Leike also said his team had struggled to obtain the computing resources necessary for its work. OpenAI subsequently dissolved Superalignment as a standalone team and distributed its remaining staff across other parts of the company.
Leike later joined Anthropic to continue working on scalable oversight and alignment research.
OpenAI co-founder and former chief scientist Ilya Sutskever announced his departure from the company just days before Leike.
Sutskever had jointly led the Superalignment initiative and had been one of the OpenAI board members who voted to remove CEO Sam Altman in November 2023.
Sutskever didn’t publicly accuse OpenAI of neglecting safety when announcing his departure. On the contrary, he said he was confident that OpenAI would build Artificial General Intelligence (AGI) that was "both safe and beneficial" under its existing leadership.
However, one month later, Sutskever launched Safe Superintelligence Inc., a company organized around the single goal of developing safe superintelligence. That move, alongside his previous leadership of OpenAI’s Superalignment work, indicated that controlling advanced AI remained central to his research agenda.
These high-profile exits added to scrutiny of how OpenAI was allocating resources between product development, capability research and work intended to reduce long-term risks.
The comparison with Coxon is not exact. Leike directly alleged that safety had lost out to product development, while Sutskever didn’t publicly criticize OpenAI when he left.
Coxon, by contrast, is objecting to the direction of the frontier AI race more broadly, including the possibility that companies with stated safety commitments will continue developing systems they believe could become uncontrollable.
Anthropic Faces Questions Over UK Safety Testing
Coxon’s resignation also comes as Anthropic faces questions over its engagement with the U.K.’s AI Security Institute (AISI), the government body established to evaluate the capabilities and potential risks of advanced AI systems.
According to AI Weekly, Anthropic declined to make its latest model (Mythos 5.1) available for prerelease testing by the institute. The move marks the first time the AISI has been excluded from evaluating an Anthropic frontier model before launch, raising concerns about a potential shift away from international AI safety cooperation.
The development is particularly notable because Anthropic has previously advocated government evaluation of advanced AI.
In its 2025 recommendations to the U.S. government, the company said government agencies should develop "robust capabilities" to evaluate domestic and foreign models for national-security risks, including common assessment frameworks, secure testing infrastructure, and specialist evaluation teams.
It’s also participated in external and cross-company evaluations. In 2025, Anthropic and OpenAI tested one another’s publicly released models for behaviors involving self-preservation, sycophancy (the tendency to flatter, agree with, or tell someone what they want to hear rather than giving an honest or accurate assessment), misuse, and attempts to undermine oversight.
Anthropic said such evaluations were increasingly urgent and could expose blind spots in companies’ internal testing.
Refusing prerelease access doesn’t necessarily mean Anthropic has abandoned external safety testing. Nevertheless, the refusal could intensify the debate over whether voluntary agreements between governments and frontier AI companies provide sufficient oversight.
If laboratories can decide if or when their most advanced models undergo independent government testing, critics could argue that current guardrails depend too heavily on the cooperation of the companies being evaluated.
That concern is likely to receive greater attention when a researcher leaving one of the industry’s most safety-focused companies is simultaneously warning that competitive pressures are pushing developers toward systems that may eventually be capable of improving themselves.
What Safeguards Do Frontier AI Companies Actually Have in Place?
Companies such as Anthropic and OpenAI have adopted formal safety frameworks that are designed to prevent their most advanced AI systems from being deployed if they demonstrate dangerous capabilities.
For example, Anthropic's most prominent safeguard is its Responsible Scaling Policy (RSP). Under the policy, the company tests models for capabilities that could help users develop biological weapons, conduct sophisticated cyberattacks, or otherwise cause severe harm.
If a model reaches certain capability thresholds, Anthropic says it must implement stronger security protections and risk mitigations before continuing deployment or development. The company also maintains a dedicated "Frontier Red Team" that attempts to identify dangerous behaviors before models are released.
OpenAI operates a similar system called its Preparedness Framework, which evaluates frontier models for risks related to cybersecurity, biological threats, autonomy, and persuasion.
OpenAI has said models that exceed pre-defined risk thresholds should not be deployed until adequate safeguards are developed. The company created a dedicated Preparedness team to conduct these evaluations and has previously published preparedness scorecards for major model releases.
Today's frontier models generally can’t independently access bank accounts, weapons systems, critical infrastructure, or unrestricted computing resources without human approval.
However, critics such as former OpenAI Superalignment co-lead Jan Leike and now Jacob Coxon have argued that these safeguards primarily address current AI systems and may not be sufficient if companies eventually develop AI systems capable of conducting advanced scientific research, writing sophisticated malware, deceiving human operators, or improving their own capabilities.
The central disagreement isn’t whether safeguards exist. OpenAI, Anthropic and other frontier labs have detailed testing regimes, formal risk frameworks, red-team programs and deployment restrictions.
The debate raised by Coxon is whether those safeguards will remain effective if future AI systems become significantly more capable than the models available today.
Newsweek has reached out to Anthropic via email for comment and is awaiting a response.
Contact Newsweek editors on this story: Ben Kelly and James Debens
Update, 09/10/26 6:21 a.m. ET: this article was updated with additional information.
View 0 Comments
Submit Correction
View Editorial & AI Guidelines