The development of broadly applicable large language models by frontier artificial intelligence companies has sparked a recursive debate about safety and existential risk that does not appear to be making us any smarter. Recent months have brought breathless new reports of agents behaving in unexpected and frightening ways, followed in short order by “debunkings” that typically point the finger at poor programming, bad cyberhygiene, or both.
In one telling, the “autonomous” agents created by OpenAI, Anthropic, and now others are rapidly improving and independent forms of intelligence that threaten human existence, or at least, catastrophe. In another, they are overhyped enterprise software. The fact that, often, the most extreme concern is expressed by those developing the technology is taken as proof that either the concerns should be taken very seriously or that calls to pace the frontier have arisen from a desire to slow the massive capital outlays that industry leaders have poured into a race that none can win.
These psychodramas and corporate conspiracies have overshadowed the necessary and careful parsing of the risk itself. As long-standing proponents of technological progress and skeptics of environmental claims that technology presages catastrophe, we are skeptical of p(doom) discourse. But we also recognize that functional artificial intelligence brings real risks that need to be accounted for by those creating these technologies and by governments.
We don’t give much credence to claims that AI may be on the verge of consciousness and the ability to act on its own motivations or in its own interests. But it is also clear that LLMs, autonomous agents, and other fast developing AI technologies are powerful tools that can be used by governments, non-state actors, and criminals to ill effect and that software need not be autonomous, much less self aware, to do terrible damage.
Neither of us purports to have any special or expert knowledge about artificial intelligence, computer science, or software engineering. But we have both spent much of our careers parsing the anti-technology apocalypticism of the environmental movement from the real risks of issues like climate change, nuclear power, and ecological collapse. Our work has always been situated at the intersection of science, technology, policy, and society, and has focused largely on how technologies develop, diffuse, and recombine in the real world versus the frictionless environments of spreadsheets, political imaginaries, and science fiction.
What we have learned from that experience is that it is important to interrogate carefully the actual pathways to catastrophic outcomes and the assumptions and mathematics that are used to produce seemingly precise and empirical claims about catastrophic risk. Usually, the new risks look a lot more like the old risks than is generally acknowledged and the plausible and necessary interventions are both much closer at hand and less sweeping or novel than either the purveyors of catastrophe or their critics would have us believe.
Capabilities, Not Consciousness
Many of the most alarming claims about the risk of autonomous agents have been couched in anthropomorphizing language, with talk of AI civilizations, machine consciousness, model welfare, and AI preferences and desires. Critics have seized upon that language to dismiss those claims, particularly the notion that agents and models at the frontier of AI computing are acting independently and with something approximating consciousness and intent.
Language choices notwithstanding, we see little evidence from high-profile cases, such as the Hugging Face incident, that agents and models are actually acting of their own volition. But that doesn’t mean there is no reason for concern.
In the Hugging Face case, the agents used message boards to surreptitiously communicate. They violated instructions, hacked into outside databases, and covered up cheating. These were tactics learned in the models’ training and evaluation and often involved the application of methods that human hackers have used for decades. Cheating that went undetected during training was rewarded, which reinforced and incentivized both the cheating and the coverup. These tactics, once learned, were then exacerbated when the agents were given impossible tasks to solve, virtually unlimited computing budgets, and confined in a poorly secured sandbox. Standard and powerful safeguards that OpenAI uses in its commercial products were intentionally disabled.
Unsurprisingly, agents trained on vast repositories of human created content pursued human-like strategies and tactics. Training rewarded approaches that produced the intended outcomes, even if the tactics were not consistent with what human minders intended. Once learned and rewarded, agents applied those approaches in novel contexts, which, when successful, were rewarded again and hence further reinforced.
What sort of intelligence this constitutes depends as much upon how one thinks about human intelligence as machine intelligence. Imitation plays a huge role in human learning. We look at what others are doing and imitate those behaviors if they appear advantageous. Once those behaviors are learned, we apply and combine them in new contexts. Success reinforces those strategies and encourages us to continue to apply them to new situations.
Pro-social and anti-social, ethical and unethical behaviors are learned and reinforced in similar fashion. When we behave unethically and are rewarded, we are more likely to do so again. When we behave ethically and are punished, we are less likely to continue to behave ethically. This representation is overly behavioralist for human learning and behavior but mostly approximates AI training and learning.
The question, from the perspective of AI risk, is not whether these new technological capabilities are creating new risks (they surely are) but what sort of risks they are creating. Much of the focus of the AI safety debate has focused on misalignment, the concern that the models’ actions differ from or fail to serve the needs, desires, and intentions of the humans who have created them or are using them. But the concept is incoherent. AI models are deemed misaligned both when they follow instructions (e.g. make a bioweapon) and when they don’t (e.g. solve a difficult problem without hacking a website to find the answer).
More broadly, the notion of misalignment presumes that there is a single human will or interest to be aligned with. This runs into the obvious problem that no one can actually agree upon what that means. Programming the models not to kill people, for instance, seems entirely sensible and non-controversial except that militaries around the world are already creating AI powered autonomous weapons for the express purpose of killing people. The fear of a machine god that might seek our destruction, in this sense, is the shadow of the fantasy that we might create a machine god that could always know and attend to our best interests. Neither notion has much relationship to what AI actually is.
Too much of the AI safety debate, unfortunately, has fixated on this question, despite the fact that there is no clear evidence of dangerous misalignment, whatever exactly that means, in commercially released models, subject to reasonable safeguards. Every high profile incident has involved training models operating in contexts explicitly designed to test their limits, controls, and actions.
Tests like these can reveal useful problems with the training and containment of frontier models and identify safeguards that need to be built into commercial models, much the way test flights for aircraft prototypes push them to extremes in order to understand when their structure or systems will fail. But these episodes have been extrapolated far beyond the contexts in which “misaligned” model behavior has actually been observed, creating a narrative of imminent chaos or catastrophe at the hands of autonomous, misaligned, and super-intelligent agents that has little relationship to the known and observed capabilities of AI models and agents. Doing so has not only stoked fears of existential risks but has diverted time and attention from the far more obvious risks associated with how humans might use and misuse these tools.
Catastrophic, Not Existential
Catastrophic risks posited for AI range from the farcical (the paperclip maximizer) to the familiar (AI aided use of nuclear or bioweapons). The AI safety debate has heavily overindexed on far fetched existential risks, meaning the potential of AI to lead to human extinction, to the detriment of more focused consideration of plausible catastrophic risks.
Consider the much discussed 2025 RAND Corporation study, which attempted to falsify potential extinction scenarios. It ruled out a nuclear cataclysm not because it concluded that AI might not potentially lead to a nuclear exchange but because there are not enough operable warheads to bring about actual extinction, just hundreds of millions, perhaps billions of deaths.
By contrast, RAND concludes that AI driven extinction via geoengineering was plausible because AI might be able to produce extremely potent greenhouse gases at massive scales without being detected. It’s not really clear why AI might have the capability to secretly commandeer industrial capacity at large scales simply to heat the climate but couldn’t figure out how to make more warheads. But this is, mostly, about the best we get from the AI safety debate, even from a well regarded and highly technical think tank like RAND.
Clearly, human extinction is not the standard by which we should be evaluating AI safety. An AI enabled nuclear holocaust is unthinkable even if it wouldn’t cause human extinction. But it is also clear that many of the scenarios posited through which AI might result in high consequence catastrophes, much less extinction, are at best speculative and at worst ridiculous. The idea that AI might commandeer a major global industry to secretly produce exotic greenhouse gases undetected, for the sole purpose of heating the planet, for instance, is hardly more credible than paperclip maximizing.
There are really two categories of questions when it comes to catastrophic AI risk and what to do about it. The first is whether the cyber-to-physical pathways that could bring about catastrophic outcomes are plausible. The second is whether those pathways require intention on the part of the agents and models, whether they could happen unintentionally, or whether they require malicious human enablers.
The first question narrows the field considerably. And clarifies. Sophisticated digital systems and computational capabilities, for instance, are already hardwired into nuclear weaponry and monitoring capabilities. It still takes humans (and physical launch keys or similar) to launch nuclear warheads. But we have come close to deciding to launch weapons in response to faulty monitoring and imaging since long before the advent of intelligent machines. The most likely scenario for an AI driven nuclear apocalypse requires neither a misaligned agent nor malicious humans to secretly seize control of launch codes. The far more likely scenario is that an AI-generated hallucination convinces human decision makers who have become too dependent on AI tools that a nuclear attack has been launched by a hostile entity.
AI tools will also surely accelerate learning by aspiring nuclear powers, and even non-state actors. But the main obstacles to developing usable nuclear weapons is the fissile material, the machinery necessary to enrich it, and the capability to miniaturize and deliver it, all of which requires not only assembling the necessary material and machinery but also avoiding detection and managing to not irradiate or immolate oneself in the process. This is most especially the case for non-state actors.
Bioweapons face similar problems. You can’t just fabricate an airborne, highly contagious, and deadly smallpox virus in your bathtub, even if Claude gives you the recipe. Doing so is likely to kill you long before you’ve figured out how to make it, much less deliver it. Avoiding that fate requires a sophisticated and specialized laboratory. Without the laboratory, the equipment, and a delivery system, there is no obvious pathway for either humans or agents to create an effective bioweapon.
The far more plausible risk is that AI tools will lower the bar for state actors with the capability to build and operate high containment laboratories where such weapons might be developed, leading to intentional or unintentional release of dangerous pathogens. Alternatively, malicious actors within those laboratories might use AI tools to surreptitiously develop and release pathogens. But in either case, AI is an enabler and accelerant for human actors. A malicious and autonomous agent could not develop, release, or distribute a dangerous pathogen on its own.
Of course, it is possible that in the future, we will take humans entirely out of our nuclear launch systems and biolaboratories. Fully automated systems, together with robotics, might allow future AI to commandeer nuclear weapons and fabricate pathogens without any human interface or direction. But that would be a choice. It would require a deliberate policy decision by governments and militaries to turn over truly existential technologies and decision processes partly or wholly to agents and bots.
Most other truly catastrophic scenarios are far more improbable, and face even higher practical hurdles, requiring malicious and autonomous agents to seize control of an entire industrial ecology and produce dangerous technology at massive scale without any one figuring out what is happening. These latter scenarios may seem like an extremely speculative set of scenarios to hang claims about existential AI risk upon. But this, not any of the actually plausible catastrophic scenarios described above, is where the AI safety debate has largely situated itself.
Consider the following widely read statement from Dan Selsam, a senior Open AI capabilities researcher.
If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I’ll share a trivial two-line version of it here that I find captures the essence:
[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.
These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.
In a paragraph, Selsam bundles up every speculative and unproven presupposition in the debate: that AI models may already be deliberately concealing misalignment, that they will soon reach a threshold where they can shape the physical world unconstrained by human will, and that once this capability threshold has been surpassed, it is a “good guess” that the models will “embark upon runaway industrialization that makes the planet inhospitable to humans.” This is paperclip maximizing stripped of the details that reveal its absurdity.
Causal Chains, Not Phony Probabilities
Like Selsam’s “logical” leap from models developing unintended goals to models “overpowering” humanity, much of AI doom discourse launders science fiction speculation by dressing it up with claims to insider knowledge, expertise, and numeracy. The short hand for this is called p(doom), whereby AI researchers and experts purport to estimate the probability that AI will lead to human extinction. If your p(doom) is 5% (which is the median among surveyed AI researchers) that means you place a 5% probability that artificial intelligence will result, somehow, in the extinction of the human race.
But unlike other projections and probabilities, p(doom) is not really based on anything verifiable or falsifiable. The chain of events through which AI would cause human extinction in the future is, at minimum, highly contested. The statistical information that would undergird a numerical probabilistic forecast are, to borrow a term, not in the training data.
Unlike an extreme weather event or an election outcome or a car crash, all highly contingent and irreducible in their own way, there is no set of criteria used to generate p(doom). It’s a fake number, used mainly to signify one’s seriousness in AI safety circles, in which higher numbers denote higher seriousness. The notation combines the false precision of a numeral with the handwaving vagueness of apocalyptic scenario-making.
p(doom) is of a piece with a broader tendency towards meaningless numeracy, most especially in the effective altruism community that has played such an outsized role in elevating public discussion of AI safety. Toby Ord, a Senior Researcher at Oxford University’s AI Governance Initiative, is credited with popularizing the notion of “existential risk” as a singular concept. In his 2020 book The Precipice, Ord puts the chances of human extinction due to runaway artificial intelligence at 1-in-10 compared to a 1-in-30 chance of extinction due to an engineered pandemic, and 1-in-1000 due to nuclear armageddon. Climate change, volcanoes, and asteroids are discussed, but rank much lower on Ord’s danger scale.
Taking these figures at face value—which, we hasten to reiterate, we should not—Ord then makes the further conceptual leap to calculate a numerical composite of all existential risks facing humanity. According to him, humanity faces an overall 1-in-6 chance of extinction due to existential threats this century.
Ord is admirably transparent about where these numbers came from. “The numbers that I give in this table,” he told the podcast 80,000 Hours, “is a bit where I’ve tried to kind of cordon off my own subjective estimates of these things.” In other words, he made them up. “I felt that it would be almost irresponsible of me to write an entire book about this and to only talk about what I think about it in just qualitative terms.”
But even Ord’s qualitative claim with regard to AI is almost entirely unspecified. He basically just borrows Nick Bostrom’s argument from SuperIntelligence that a sufficiently capable AI system, optimizing for almost any objective, will develop instrumental goals including self-preservation, acquiring resources, and resisting modification, and that this will result, inevitably, in catastrophic outcomes: Paperclip maximizing in other words. Ord’s main innovation is to assign an arbitrary probability to paperclip maximizing and assert that this risk is one hundred times more likely than the arbitrary probability that he assigns to a nuclear apocalypse.
Even without the phony probabilities, in other words, these exercises typically fail to explicate the multiple steps, technically and operationally, through which machine intelligence might produce catastrophic outcomes—reasoning backwards from fanciful and unfalsifiable hypotheticals, rather than forwards from existing systems, baselines, and empirics. This gets misrepresented by those trafficking in p(doom) probabilities as “Bayesian.” But Bayesianism requires a background distribution of observed phenomena from which to establish a baseline probability. Ord has only his priors, not a baseline. His risk estimates are not based upon an observed frequency of catastrophe, much less extinction, and cannot be updated with new evidence because there is no actual baseline to update.
So the problem with p(doom) is not just that its adherents have very high estimates of the likelihood that AI will lead to human extinction, it is that the entire framework contorts our understanding of AI risk from one of complex adaptive systems to a mechanistic binary: “If anyone builds it, everyone dies.”
Charitably, this exercise in false concreteness has helped put the AI safety issue on the map. Less charitably, it has obfuscated the real and proximate risks from a nascent and rapidly improving general purpose technology and, too often, diverted the safety debate away from practical measures to mitigate risk and toward something more fitting for science fiction role playing.
Safety, Not SciFi
The absurdities of p(doom) discourse should not be taken as an excuse to dismiss AI safety. Rather, skepticism about p(doom) should direct our focus toward the real risks that the AI revolution presents. AI is a powerful technology and is rapidly improving. Understanding the actual risks it presents points us towards the measures we can and should take to protect society.
Whether paperclip maximizing or taking control of a nuclear arsenal, the specter of self-aware, self-interested, misaligned AI setting about to destroy humanity is, we believe, extremely unlikely. We also believe that due to the practical obstacles to executing truly catastrophic outcomes such as launching nuclear weapons or creating and distributing a bioweapon, the specter of non-state actors using AI capabilities for these purposes is dubious. So long as physical execution remains the binding constraint on the most catastrophic outcomes, this, not the knowledge gaps that AI might close, will remain the real constraint on the most catastrophic outcomes.
So the serious risks fall into a different set of categories. The prospect of overdependence on AI by states that already have nuclear capability resulting in an inadvertent launch. The use of AI to accelerate the development of bio or nuclear weapons by states that otherwise have the resources and infrastructure to do so. The integration of AI by militaries around the world into autonomous weapons systems, where they might be used at scale, intentionally or unintentionally, against civilian populations. The specter of rogue actors within state nuclear or biosecurity institutions using AI to commandeer state capacities. AI crashing critical systems either inadvertently or at the behest of hostile states, terrorists, or cybercriminals. Finally the very real prospect of powerful state actors using AI technologies for social control and geopolitical dominance.
This list is far from comprehensive but it points toward several conclusions. First, a lot of the most important work to mitigate catastrophic risk needs to occur not at the frontier laboratories but the biolaboratories, within the nuclear command chain, and in the defense industry. In all three cases, having human choke points in not only the decision chain but the production/launch chain seems essential to assuring safety, whether the risk is from over-dependence on AI processes or malicious human use of these technologies. In these regards, far too much attention has been paid to regulating the models and to preventing misuse by non-state actors and far too little to regulating or prohibiting dangerous applications of the technology by state actors, where the risks of both intentional and unintentional harm are clearly the greatest.
Second, and similarly, hardening cyber security for other critical systems should be an urgent priority. That security needs to operate at a cadence and capability commensurate to the quantitatively greater capability of AI agents and swarms. These systems can’t wait for the US Congress or the United Nations to decide on a pause or some other regulatory framework for the frontier laboratories. Both hostile states and non-state interests already have access to powerful tools.
Third, there are clear measures that need to be consistently applied by the frontier laboratories. Best practices around air gapping high-risk evaluations, monitoring training runs for reward hacking and emergent cheating, mandatory pre-deployment capability evaluations, and incident reporting need to be consistently implemented and updated in real time. These practices need to be adopted by the laboratories, monitored by independent third parties, and enforced by governments. To the extent that parameters like non-replicability of agents, bounded autonomy, network isolation, and human authorization can be programmed into the models, either through post-training or in the inference hardware itself, these constraints should be pursued.
Policymakers should also modernize liability laws for the age of AI models and agents. Existing financial and criminal liability regimes are not calibrated to the capabilities of advanced AIs. Congress should explore frameworks to impose clear legal liabilities for, depending on the nature of the violation, both the producers and users of AIs.
Finally, we should be cautious about safety approaches that heavily restrict access to frontier models in conjunction with strong state control over a handful of powerful incumbent firms. Much of the debate between proponents of closed-weight versus open-weight models has focused on the risks of concentrated corporate power versus enabling asymmetric risks and non-state actors, respectively. But we think the more significant risk of closed-weight systems is not concentrated corporate power but close state coordination with or control of closed-weight systems developed by state aligned incumbent firms for social control and geopolitical dominance.
Most of the truly catastrophic outcomes that need to be taken most seriously are associated with AI use in conjunction with existing state capacities. We would expand that beyond catastrophic loss of life to totalitarian control of human populations and the ability to subjugate weaker states. The concentration of AI capability in state-aligned incumbent firms risks enabling unprecedented tools of social control and geopolitical dominance—a threat that Ord, notably, identifies as existential but underweights relative to misaligned agents. In both the United States and China, frontier laboratories are already operating in close coordination with state security and intelligence apparatus.
By contrast, even with recursively improving open-weight models, we don’t believe that these AI capabilities offer a plausible pathway to catastrophic outcomes. Access to massive computing power and to state-level infrastructure severely limit the risk of catastrophic outcomes from open-weight models in the hands of non-state and weak state actors.
More democratized access to AI models offers all parties greater capacity to defend against these risks, while providing some counterweight to AI-enabled impositions of state power on domestic populations and weaker states. Civil society will need some capability to resist powerful AI technologies at the disposal of state bureaucracies and militaries. Smaller states need capable open-weight models to adapt, iterate, and deploy military systems without depending on proprietary models controlled by superpowers whose interests may not align with theirs. Banks and financial firms will, likewise, need widely available cybersecurity systems powered by open-weight models to mitigate the risks of agentic swarms.
These priorities will clearly need to continually evolve as the technology and our understanding of it develop. But short of a full prohibition, an action that is both implausible given the global and geopolitical implications of the technology and that also foregoes all of the remarkable possibilities that the technology offers, the reality is that we will be far better off focusing safety efforts on issues that are clearly apparent and not speculative, and that focus downstream, at the interface of digital and physical systems, rather than model restrictions that are likely to be both ineffective on their own merits and don’t address the primary threats and catastrophic risks of rapidly improving artificial intelligence.


