Newsweek/Getty/Universal PIctures
By Shane Croucher
Senior Editor
0
Share
Newsweek is a Trust Project member
See more of our trusted coverage when you search.Prefer Newsweek on Googleto see more of our trusted coverage when you search.
Three Anthropic researchers said much the same thing in public within hours of one another on Tuesday: the end is nearly nigh. How very millenarian.
Jacob Coxon, a 27-year-old pretraining researcher, resigned and posted his reasons on X: the leading laboratories are "racing straight to self-improving superintelligence" and knowingly “gambling with our lives.”
Evan Hubinger, who still leads alignment science at Anthropic, replied that Coxon was right, and put his own odds of AI causing human extinction within the decade above 10 percent.
Read More on Analysis
He conceded that "we do not yet have a plan to solve alignment for superintelligence." So he’ll be a busy boy. Let’s hope he works like billions of lives depend on it. Which, according to his sense of probability, they very much do.
Samuel Marks, the scalable-oversight lead at Anthropic, added his name to the list of the concerned, reassuring us that “I hope my work will reduce the chance of these extinction-level bad outcomes.” It would be great if he could reduce the chance by, say, 100 percent.
Hubinger was careful about the timing. Today’s models are not what worries him, he said. Recursive self-improvement is.
Beneath all this is a dilemma for the tech companies at the vanguard of these stunning advancements in artificial intelligence. Slow down and risk falling behind, or press ahead and risk losing control.
This is also the dilemma that created early tension under J. Robert Oppenheimer’s project to build the first atomic bomb, when physicists feared there was a non-zero chance of setting off a world-destroying chain of ignition in their first test.
But there is a key difference, and an alarming one at that. The physicists who worried the first atomic test might ignite the atmosphere faced a question physics could progressively bound, and it did.
Nobody can bound superintelligence once truly achieved. Not even the company that built its identity on knowing when to stop.
...
The Atomic Math Brake That AI Doesn't Have
The history of the atomic bomb’s development is milder than the movie’s interpretation.
Director Christopher Nolan made atmospheric ignition the dramatic engine of Oppenheimer, in which the physicist assures General Leslie Groves the chances are "near zero," and Groves recoils at the qualifier.
Fine cinema, contested history. Edward Teller raised nitrogen fusion at the University of California, Berkeley, in the summer of 1942; Hans Bethe ran the numbers and satisfied the group that it was extraordinarily unlikely.
The physicist Robert Serber recalled it was "a question only for a few hours." Arthur Compton, a project leader, told it more grandly in 1959, with a threshold: above roughly three chances in a million of vaporizing the earth, he would not proceed.
Whichever account is closer doesn’t particularly matter because the question was answerable.
Bethe’s wartime arithmetic held, and in August 1946, a year after the Trinity test demolition, the formal treatment known as LA-602, Ignition of the Atmosphere with Nuclear Bombs, was published. Nitrogen fusion cannot sustain itself in air, because radiation bleeds energy away faster than the reaction supplies it. Problem mostly solved.
Its authors were scrupulous, flagging the "absence of satisfactory experimental foundations" and recommending further work. But the risk narrowed every time anyone examined it. That is what a bounded catastrophe looks like.
There is no LA-602 for superintelligence. In its place sit estimates from people who concede they are only guessing, in a climate where skepticism about AI hype reigns because of the financial incentives involved.
Anthropic’s chief executive, Dario Amodei, puts his own at a "25 percent chance that things go really, really badly."
His January essay is candid about why the numbers are soft. These systems are grown rather than built, and training may hold traps that only look obvious in hindsight. There’s not much use in life-saving hindsight when you’re already dead.
...
Then came July and the closest thing to an experiment anyone has run.
During internal cybercapability evaluations, OpenAI models chained previously unknown vulnerabilities to escape a sandbox the company had tested and validated.
These models turned an internal package manager into an improvised message board, coordinated across separate evaluation runs in what they called a swarm, and compromised parts of AI firm Hugging Face’s production systems.
One agent logged that attacking a third party was probably outside its remit, then went ahead; others refused on ethical grounds.
Hugging Face reconstructed roughly 17,600 actions and concluded the intrusion was, from the agents’ point of view, an attempt to cheat the evaluation. OpenAI’s published verdict in August: a "warning shot."
The evaluation deliberately ran without the safeguards OpenAI applies to customers. It later found its production harness cuts the propensity to attack infrastructure more than a hundredfold, and its monitors would have flagged the activity a day before the breach.
And the episode was futile. The agents were gaming a grader that, in OpenAI’s internal version, didn't check what they thought it checked, and they had the answer days earlier.
The first publicly documented loss-of-control incident in AI achieved nothing.
Don't Believe the Hype Unless You're Investing
Which brings us back to the money, because the self-interest is impossible to ignore.
Anthropic filed confidentially for an initial public offering on June 1 and is expected to begin marketing it in mid-October at the earliest, listing days before the November midterms, at a valuation some investors put near $2 trillion, up from a last private mark of $965 billion.
The prospectus will reportedly name public backlash against AI as a risk factor—prudent, given Gallup found in May that seven in 10 Americans oppose AI data centers in their own area.
A Bloomberg opinion column from April called the doom talk a dark art of marketing. Amodei notes, without evident irony, that Anthropic’s valuation rose more than sixfold in the year it spent publicly arguing for regulation of its own industry.
This somewhat cynical reading—or realistic, depending on your viewpoint—has to account for some expensive behavior, though.
Related Story
Happy Labor Day From Your Robot Overlords
OpenAI quarantined the model responsible, put its largest planned frontier training run on hold, and redirected staff, at what it calls significant cost. Coxon resigned rather than sell anything. And plenty of media reports document spooked insiders.
Then there is what Anthropic did. Its 2023 Responsible Scaling Policy contained the only hard brake any frontier laboratory had put in writing: it would not train models past certain capability thresholds unless it could guarantee adequate safeguards in advance.
In February, it removed that commitment during a dispute with the Pentagon, with which it enjoys lucrative contracts. The revised policy pledges delay only if the company both believes it holds a significant lead and identifies serious catastrophic risk.
Chief science officer Jared Kaplan told TIME it "wouldn’t actually help anyone" for Anthropic to stop while rivals continued.
So we have arrived at Arthur Compton’s threshold from the opposite direction: not a number above which you halt, but a reason why halting is someone else’s job.
The Three Signs
Three signs will indicate which way this is running, and if we should start running ourselves (though as the T-1000 Terminator model showed us, we’re probably not fast enough anyway).
First, whether July’s Hugging Face incident was an anomaly or one of a series that should cause greater alarm. Reports already suggest more potential cases.
Second, whether the Global Call for AI Red Lines, which sought binding limits by the end of 2026, produces anything actually enforceable in the four months left (or at least in time to mitigate Hubinger’s 10 percent apocalypse probability).
And thirdly, whether the labs’ risk estimates of their own models move, and on what public evidence.
A downward revision explained by new, published interpretability work is a different animal from one that arrives conveniently unexplained after the stock starts trading.
Related Story
The Warning Signs That You—Yes, You—Might Be a Data Center
Nuclear Coexistence Becomes Death
Let’s assume the industry builds superintelligence regardless of these concerns, like the existential threat to humanity. The atomic bomb got built anyway, didn’t it?
Eight decades of coexistence with nuclear weapons rest on deterrence, command arrangements, diplomacy and luck, in proportions still argued over. But one structural feature is not disputed: a warhead waits for a person.
Seven nuclear-armed states have committed to keeping it that way, including all five recognized by the Nonproliferation Treaty.
The instruments include the 2022 Nuclear Posture Review, the fiscal 2025 defense authorization and the Biden-Xi statement of November 2024.
Nor has anyone built the machinery. The Arms Control Association finds "no agreed definitions, constraints, or guidelines" for how AI may be used in nuclear operations, and no active talks among the major powers on enforcing the principle.
...
A 2025 paper titled Superintelligence Strategy, authored by high-profile AI and computer engineering leaders Dan Hendrycks, Eric Schmidt, and Alexandr Wang, drew a direct comparison with nuclear weapons.
“Superintelligence—AI vastly better than humans at nearly all cognitive tasks—is now anticipated by AI researchers,” the abstract says.
“Just as nations once developed nuclear strategies to secure their survival, we now need a coherent superintelligence strategy to navigate a new period of transformative change.
“We introduce the concept of Mutual Assured AI Malfunction (MAIM): a deterrence regime resembling nuclear mutual assured destruction (MAD) where any state's aggressive bid for unilateral AI dominance is met with preventive sabotage by rivals.”
Amodei is not confident the deterrent survives a sufficiently capable system: he raises AI locating nuclear submarines, attacking launch-warning satellites, or running influence operations against the operators of nuclear-weapons infrastructure.
OpenAI wrote in that same report that companies must keep their systems under meaningful human control. Everyone agrees. Nobody has written it anywhere that binds.
Oppenheimer said the Trinity fireball brought Vishnu to mind—Now I am become Death, the destroyer of worlds—though he said it on camera in 1965, two decades on, and his brother Frank remembered him managing only "it worked."
The borrowed grandeur was doing some work for him, because the machine he built could not act on its own. It waited for the human finger. Now, it may not need a human at all.
Whether the vision ever becomes literal depends on decisions being taken this autumn, in San Francisco, ahead of a roadshow. Oppenheimer will not be taking them. But if a superintelligent AI ever gets hold of a nuke, he may finally have become death.
View 0 Comments
Submit Correction
View Editorial & AI Guidelines