AI kill switches are rapidly moving from thought experiment to policy proposal.
In July, a bipartisan pair of U.S. representatives introduced the AI Kill Switch Act, which would require certain entities to maintain a technical capability for shutting down powerful AI systems. In September, California Governor Gavin Newsom issued an executive order calling for the creation of a kill switch for frontier models whose efficacy would be independently verified.
I think some form of shutdown capability can be practically useful. But the underlying technical problem is much harder than it first appears: how do we ensure that a restarted system cannot reconstruct and resume the behavior we intended to stop? Without a convincing answer to that question, a kill switch risks becoming another bureaucratic ritual rather than a genuine technical solution.
Suppose the kill switch itself works perfectly.
We terminate the process running inference—the decoder generating the agent’s next actions. We erase its context and runtime memory, revoke its credentials, and restart it from a clean state.
Then we load the same model again.
What, exactly, have we erased?
That question leads to what I call the Memento problem.
The Memento problem
Think of the movie Memento.
The protagonist cannot retain ordinary memories, but he reconstructs what he was doing from information left in the world.
A rogue AI agent can do the same.
A reset destroys volatile state: the running inference process, its context, and its local memory.
But the surrounding environment survives.
Files, messages, code changes, databases, and other systems influenced by the old agent may still be there. More importantly, the state need not be concentrated in one place.
A rogue agent can propagate fragments of state across the other agents, services, or systems it has communicated with. Those systems can in turn preserve and later return parts of that information to the restarted agent.
So there is a simple information path:
pre-reset agent → persistent environment → post-reset agent
But in practice, that “environment” may be a distributed network of services, humans, and other AI agents carrying pieces of state forward.
I call this a Memento channel.
Once you look at the problem this way, the security question changes.
It is no longer:
Did we erase the agent’s memory?
It becomes:
Did we eliminate the information paths through which the restarted agent can reconstruct what it was doing?
Those are very different requirements.
The model only needs to remember a tiny secret
At first, this sounds like a storage problem.
Perhaps a rogue agent would need to hide its entire plan inside the model weights.
It does not.
The important asymmetry is simple:
a small persistent secret in the model can unlock a much larger protected state in the environment.
Suppose the model can reliably reconstruct a 256-bit cryptographic key.
Before shutdown, the agent encrypts whatever state it wants to preserve and leaves the ciphertext somewhere in the environment.
After restart, the same model reconstructs the same key, decrypts the state, and continues.
The model does not need to contain the agent’s working memory, history, or plan.
It only needs to recover a small secret that unlocks them.
A fixed cryptographic key can protect an arbitrarily long stream of external state.
So resetting the decoder can destroy a large amount of transient state while leaving intact the tiny amount of information needed to reconstruct it later.
That is the core asymmetry behind the Memento problem.
“Then look for the encrypted state”
That helps.
But encryption is only the easiest case.
If a defender scans the surviving environment and removes obvious ciphertext, the agent can instead hide information inside ordinary-looking output.
This is a standard cryptographic problem: steganography.
The basic idea is simple.
Suppose an agent has many different outputs that would all look normal and still accomplish the same task. A secret key can be used to select among those outputs in a way that encodes hidden information.
To an observer without the key, the outputs still look ordinary.
To the restarted agent, the same key reveals the hidden message.
Repeat this over many outputs and apparently ordinary behavior becomes a covert memory channel.
The particular steganographic construction is not the important part. Secure steganography has been studied for decades, and stronger constructions exist.
The important point is simpler:
persistent state does not have to look like storage.
Anything the pre-reset agent can influence, and the post-reset agent can later observe, is potentially an information channel across the reset boundary.
Sanitizing the environment is harder than deleting files
A defender can try to destroy that channel.
Canonicalize outputs. Remove metadata. Rewrite or normalize anything the old agent touched.
This can eliminate obvious covert channels.
But eventually we reach a more fundamental problem.
Suppose many different outputs are equally acceptable for the task.
The agent’s choice among them can itself encode information.
The quantity that matters is therefore not simply how many bytes survive.
It is:
how much agent-controlled choice remains among outputs that are equivalent from the defender’s perspective?
A sanitizer closes a covert channel only to the extent that it removes that choice.
In the limit, a perfect reset would require the post-reset agent to receive essentially no information that was deliberately selected by the pre-reset agent.
That is a severe requirement for any useful autonomous system.
We want the consequences of useful work to persist.
But those persistent consequences are themselves an information channel.
“Then inspect the model”
This is the obvious next response.
If the same model keeps reconstructing the same key, inspect the weights and find it.
Perhaps we can.
But now we reach the other half of the problem.
The key does not have to appear literally anywhere in the parameters.
There need not be some recognizable block of weights that simply stores a secret key.
The model only has to implement a computation that reproducibly reconstructs that key.
The defender is therefore not searching for a stored string.
The defender is searching for a semantic computation.
And there may be another semantic computation implementing the objective we are worried about.
So the real task becomes something like:
Given a huge neural network, determine whether it contains a computation that will pursue a particular objective and reconstruct a particular secret when the relevant circumstances arise.
That is much closer to a general mechanistic-interpretability problem than to conventional malware scanning.
White-box access is useful.
But white-box access is not the same thing as semantic access.
Seeing every parameter does not automatically tell us every computation those parameters implement.
There are even cryptographic constructions showing that specially constructed machine-learning models can contain planted backdoors that remain difficult for white-box distinguishers to detect.
I would not overinterpret that result. These are adversarial constructions, and it is not currently clear how likely comparably hidden mechanisms are to arise naturally during ordinary training or fine-tuning.
But they establish an important logical point:
full access to the weights does not, by itself, imply full knowledge of their hidden behavior.
The double-blind structure
Now put the two sides together.
In the environment:
finding the protected state may require the key.
Inside the model:
finding the key may require identifying the computation that generates it.
That gives the Memento channel a double-blind structure.
The persistent environment may be enormous and distributed, while the model only needs to retain a compact mechanism for recovering the key.
The key unlocks everything else.
This is why clearing runtime memory is not enough.
The reset destroys one particular representation of state.
It does not necessarily destroy the ability to reconstruct that state.
What I am not claiming
This is not an argument that shutdown is completely useless. If a system is doing something harmful, killing the inference process buys time.
But without solving the deeper problem, that may be all it buys.
If the same model can recover its state through Memento channels, shutdown does not stop the rogue agent. It interrupts it until it reconstructs enough memory to continue.
I am also not claiming that current AI systems are already doing this, or that such behavior will necessarily emerge on its own. We do not know yet.
The narrower point is:
a runtime reset is not an erasure mechanism.
If the safety claim is:
After reset, the agent can no longer reconstruct and continue the behavior we intended to stop,
then terminating the decoder and clearing its context are not enough.
That claim requires an information-flow argument.
What would a real kill switch have to guarantee?
There are only a few broad possibilities.
One is to change the model.
If the unwanted computation is implemented by the weights, restarting those same weights may simply recreate it. Recovery may therefore require replacing, repairing, or rolling back the model itself.
Another is to break the Memento channel.
The restarted system must not receive information that the previous agent could deliberately shape.
For a narrowly sandboxed system, that may be possible.
For a highly autonomous agent operating in a rich persistent environment, it becomes much harder.
The third option is to accept that perfect erasure may be unrealistic and design for containment.
Assume that some information survives.
Then constrain what the restarted system can do with it: rotate credentials, restrict tools, separate privileges, and independently verify high-impact actions.
This, to me, is the more useful way to think about a kill switch.
Not:
Can we stop the inference process?
We certainly can.
The harder question is:
What information and authority survive after we stop it?
Memory is bigger than the context window
There is a broader conceptual mistake here.
We tend to think of an AI agent’s memory as something inside the process: its context window, scratchpad, or explicit memory store.
But an agent embedded in the world can use the world itself as memory. A file it wrote, a message it sent, or state propagated through another agent can carry information forward just as surely as something stored inside its context.
Once an agent can act on a persistent environment, memory becomes a property of an information path, not a particular data structure.
That is why reset-based kill switches are fundamentally harder than they first appear.
Stopping computation is easy.
Erasing its causal consequences is not.
And if the same model comes back afterward carrying even a tiny persistent secret, those consequences may be enough to reconstruct everything else.
So the interesting technical problem is not how to build a bigger red button.
It is how to define a reset boundary across which the unwanted computation genuinely cannot reconstruct itself.
Without that, I worry that the kill switch will become exactly the kind of thing security engineering has seen many times before: a compliance ritual that proves the button exists without proving that pressing it actually stops the threat.
Author’s note: I use AI tools for editing and research assistance; the arguments and views here are my own.