“AI could kill us. Slow down development. Make sure humans can always shut it off.”
That is increasingly the framing of the frontier-AI safety debate. Anthropic CEO Dario Amodei has called for pacing frontier development; OpenAI CEO Sam Altman has supported that direction. Nvidia’s Jensen Huang and Meta’s Mark Zuckerberg have publicly rejected coordinated slowing. So even among the largest AI companies, there is no consensus on whether pacing is workable.
But I think the debate starts one level too high.
Before asking whether we should slow AI, pause AI, regulate AI, or build a kill switch, there is a more basic question:
What exactly is the threat model?
This is basic security engineering. You do not design the defense before deciding what failure you are trying to prevent.
AI-assisted cyberwarfare is one threat. AI-assisted biological misuse is another. An agent taking unauthorized actions through tools is another. A highly autonomous system that can acquire resources, improve itself, replicate, and resist shutdown is another.
These are not one problem called “AI risk.” They have different assumptions, attack surfaces, and defenses.
And if we collapse all of them into “AI might kill everyone,” we risk building very expensive defenses against a vaguely specified enemy.
Start with the threats we can actually describe
The catastrophic scenario I find most plausible in the nearer term is not an AI spontaneously deciding to wipe out humanity. It is humans using AI as a force multiplier.
Cybersecurity is the cleanest example.
OpenAI recently classified GPT-6 Astra at its “Critical” cybersecurity capability level. In OpenAI’s description, given the appropriate tools and access, the model can find previously unknown vulnerabilities and develop ways to exploit well-protected systems without a human guiding every step.
That is already a meaningful change in the threat model.
Now project forward. Give systems like this longer-running agents, network access, exploit-development tools, and a malicious operator. The concern is not that the AI has suddenly developed hatred for humanity. The concern is automated cyber operations running at machine speed: vulnerability discovery, exploit generation, lateral movement, adaptation to unfamiliar systems, and potentially attacks on critical infrastructure.
Biology has a similar structure. The danger is not necessarily that an AI independently decides to engineer a pathogen. The more plausible concern is that AI lowers the expertise barrier for someone who already has intent, helps with design and troubleshooting, and compresses work that previously required a larger team—provided that person also has access to a competent wet lab and can execute the biology in the real world.
That last constraint matters. AI does not magically eliminate physical bottlenecks.
Then there is the more speculative scenario: a highly autonomous AI becomes good enough at AI research, operates infrastructure, acquires resources, replicates, and resists attempts to shut it down. Anthropic explicitly tracks automated AI R&D as one of the capability thresholds in its Responsible Scaling Policy.
I do not think this scenario should be dismissed. But notice how many conditions have to line up. The system needs very high capability, long-horizon autonomy, access to important resources, the ability to act outside controlled environments, and enough operational resilience to survive intervention.
That is a legitimate threat model.
It is not the same threat model as a criminal using an AI system to compromise a hospital network.
We should not pretend one defense handles both.
Kill-switch thinking starts too late
This is why I find the “kill switch” framing unsatisfying.
A kill switch may be useful. But if my entire safety architecture depends on pressing a button after an AI system has already acquired broad access, credentials, autonomy, and control over important infrastructure, then something went wrong much earlier.
The more basic question is:
Why did we give one system that much authority?
Computer security has spent decades learning the same lesson. Do not give every process root access. Do not expose every system to every network. Separate privileges. Isolate components. Constrain interfaces. Monitor actions. Keep independent recovery mechanisms.
AI does not invalidate any of that.
If an agent needs access to three databases, give it access to three databases—not the entire corporate network. If it writes code, execute the code in a sandbox. If it wants to perform a high-impact action, require a separate authorization step. If it handles sensitive information, constrain where that information can flow. If it invokes tools, restrict the tools and arguments it can use. If it generates a patch, verify the patch before deployment.
These are mundane controls.
That is fine.
Security is mostly mundane controls applied correctly.
Some current safety work is moving in the right direction
There are already useful examples of this more concrete approach.
OpenAI’s response to Astra’s cyber capabilities included stronger isolation, encrypted checkpoints, monitoring of full agent trajectories, and blocking evaluations before internal use. Anthropic’s Responsible Scaling Policy similarly ties particular capability thresholds to stronger safeguards, security requirements, external review, and risk reporting. Its current roadmap includes projects such as anomaly detection and mechanisms for cryptographically or otherwise provably attributing outputs to particular model weights.
I do not regard any of these frameworks as complete solutions. The point is not that frontier labs have validated the argument.
The useful part is the structure:
identify a capability, identify the failure it enables, then build a specific control around that failure.
That is much closer to security engineering than saying “AI is dangerous, therefore slow down.”
Why I am skeptical of “pacing” as a long-term solution
There are real bottlenecks in frontier AI today. Large-scale compute is the strongest one. Frontier training requires advanced accelerators, high-bandwidth memory, networking, power, cooling, and large data centers.
So yes, restricting enormous training clusters can slow development at the very top.
But then come two obvious questions:
Restricted by whom? And for how long?
Compute is spread across countries and companies. Hardware improves. Algorithms become more efficient. Smaller models become stronger. Fine-tuning and reinforcement learning extract more capability from existing weights. Inference-time search and tool use can make an existing model substantially more capable without repeating the original training run.
So “control the GPUs” is not the same as “control AI capability.”
Data is not an obviously durable chokepoint either. High-quality data and expert feedback matter, but in many technical domains the more interesting bottleneck is becoming verification rather than data generation.
For mathematics, theorem provers such as Lean can check candidate proofs. For software, compilers, tests, fuzzers, static analyzers, and formal methods can provide automated signals about whether an output is correct. Once answers can be checked automatically, a system can generate large amounts of its own training experience without paying humans to label every example.
Tacit engineering knowledge is probably an even weaker long-term barrier.
A lot of current AI development is an empirical loop: propose something, implement it, run the experiment, inspect the failure, modify it, repeat. AI systems are increasingly good at exactly those activities—reading papers, generating code, debugging experiments, constructing evaluations, analyzing logs, and suggesting the next experiment.
What currently looks like scarce institutional know-how may therefore become reproducible surprisingly quickly.
And then there are model weights.
Before release, access to weights can be controlled. Once strong weights are broadly distributed, the economics change completely. You no longer need to reproduce the original expensive training run. You inherit the result, then fine-tune it, distill it, quantize it, give it tools, generate synthetic data from it, or train specialized agents around it.
That is the asymmetry I think matters most:
creating a capability is expensive; reproducing and adapting it after it has spread is much cheaper.
What if the largest labs really do slow down?
The absolute frontier probably slows temporarily.
That is not controversial. The largest labs have compute, infrastructure, data pipelines, and experimental scale that smaller groups cannot easily match.
But progress elsewhere does not stop.
Open models keep improving. People fine-tune them. Others distill them. Tooling gets better. Synthetic data improves post-training. Inference-time techniques improve. AI itself helps smaller teams reproduce techniques that previously required much larger groups.
There could even be a snowball effect. If the largest labs slow while strong models and methods are already available, researchers, investment, and experimentation may shift toward the open-source ecosystem. More contributors improve the tools; better tools lower the barrier for the next group; capability development becomes increasingly distributed.
The current industry disagreement already illustrates the coordination problem. Amodei has argued for pacing, while Huang and Zuckerberg are explicitly rejecting coordinated slowing.
If the major companies cannot agree on slowing now, I would not build a long-term safety strategy on everyone coordinating indefinitely.
There is also a useful historical analogy. The U.S. spent years trying to restrict the export of strong cryptography. Those controls imposed real friction for a period, but they did not keep strong encryption globally scarce. AI is different—large-scale compute is much more physically concentrated—but the general lesson still matters: once technical capability is widely distributed, containment becomes much harder.
The more interesting strategy is to automate defense
This is the part of AI safety I think deserves much more attention.
If AI makes offense cheaper, use AI to make defense cheaper.
If AI writes code, automatically verify more of that code. If agents use tools, automatically constrain and monitor those calls. If models process private data, track where the information flows. If jailbreaks change, automate jailbreak discovery and hardening. If AI generates patches, use tests, static analysis, formal methods, and other AI systems to check them before deployment.
This is not a grand solution to “alignment.”
It is something more useful: a collection of concrete, testable safety properties.
Can this agent read this file?
Can it send this data outside the organization?
Can it execute this command?
Can it modify production?
Can its generated patch violate this invariant?
Can we detect when its behavior leaves the allowed operating region?
These are questions we can build systems around.
And importantly, defense does not always require solving the full intelligence problem. You do not need to know what a model “really wants” to stop it from accessing a database. You do not need perfect interpretability to prevent an unauthorized transaction. You do not need a theory of consciousness to sandbox generated code.
Some of the safeguards are considerably easier than the problem we are trying to solve around them.
That asymmetry is worth exploiting.
Regulation should measure outcomes, not paperwork
This is also why I worry about compliance theater.
Hire 50 lawyers. Produce hundreds of pages of documentation. Run the mandatory audit. Check every box.
Then somehow conclude the system is safer.
In raw technical terms, none of that by itself reduces the risk.
We have seen this tendency with regimes like GDPR: lots of process, lots of compliance machinery, and often a weak connection between the paperwork and whether the underlying technical harm actually went down.
AI regulation can easily fall into the same trap.
I would rather regulate things we can test. Can an agent cross its authorization boundary? Can private information leak? Are dangerous tool calls constrained? Can generated code execute without verification? Can anomalous behavior be detected and contained?
If a company says its system is hardened against prompt injection, test it. If it says private data cannot leak, measure the information flow. If it says generated code is safe, verify the code.
Compliance should demonstrate safety. It should not become a substitute for safety.
Move the debate down one level
The AI safety debate is dominated by large words: extinction, alignment, pause, acceleration, regulation, open source, kill switches.
Those words are useful for headlines.
They are not threat models.
A serious technical safety program should move down one level. What can the system actually do? What can it access? Who controls it? What does an attacker need? What concrete event causes harm? What signal tells us that event is happening? What mechanism prevents, detects, or contains it?
Then build that mechanism.
I am not arguing that catastrophic AI risk is imaginary. I am arguing that panic is not a security architecture.
And I am highly skeptical that long-term AI safety can depend on keeping advanced capability scarce forever.
The technical race I care about is different: as AI gets better at coding, attack, experimentation, and automation, can we make verification, monitoring, containment, hardening, and defense improve just as quickly?
That is where I think a serious AI safety agenda should begin.
Author’s note: I use AI tools for editing and research assistance; the arguments and views here are my own.