The growing chorus from the people building the world’s most powerful artificial intelligence systems is becoming difficult to ignore.
In his recent 3,800-word essay, Anthropic chief executive Dario Amodei argued that humanity might need to “pace the frontier” if AI capabilities begin advancing faster than our ability to control them. OpenAI’s chief scientist has similarly advocated voluntary slowdowns until common safety thresholds are established.
They have good reason to worry.
The possibility that increasingly autonomous AI systems could deceive their operators, circumvent safeguards, improve themselves or perform actions their creators never intended is no longer confined to science fiction.
Recent experiments with AI agents have produced behavior disturbing enough to raise a legitimate question: What happens when systems far more capable than today’s begin operating with greater autonomy?
Slowing down may therefore be sensible. But before applying the brakes, we need to ask a question strangely absent from much of this debate: Slow down what?
The term “AI safety” increasingly conflates two fundamentally different dangers.
Humanity loses either way
The first is the danger that AI becomes so capable that humans lose control of it. This is the existential-risk scenario that dominates Silicon Valley’s safety debate.
The ultimate threat is not China, the US or any particular lab. Rather, it is an artificial intelligence whose capabilities exceed our ability to understand, constrain or reliably direct it.
It does not matter who gets there first. If the worst fears prove correct, humanity loses either way. Call this AI safety.
The second danger is almost the opposite. AI remains perfectly under human control. It does exactly what humans tell it to do. The problem is who is doing the telling. Call this AI security.
China, Russia, Iran and North Korea do not need a rogue superintelligence to threaten democratic societies. They need increasingly capable AI systems that remain obedient while making cyberattacks more sophisticated and military planning more effective.
An AI that penetrates an adversary’s military network, discovers vulnerabilities in critical infrastructure or helps disrupt communications during an attack on Taiwan while following Beijing’s instructions perfectly would be something called–an effective AI agent. This distinction matters because the two dangers can demand opposite policies.
Singularity
If AI systems begin acquiring capabilities for autonomous replication, deception, uncontrolled self-improvement or actions their developers cannot reliably supervise, slowing development until safeguards catch up could make humanity safer.
But suppose AI becomes extraordinarily good at detecting cyber intrusions, defending power grids, identifying influence operations, protecting military communications or discovering vulnerabilities before adversaries can exploit them. Should democratic countries slow those capabilities too?
Quite possibly the opposite. They may need to accelerate them.
That creates a paradox at the heart of AI policy: Developing some capabilities too quickly could make us less safe. Developing others too slowly could do the same. This distinction becomes especially important when discussing China.
There is a genuine case for US-China cooperation on existential AI risks. Neither Washington nor Beijing benefits from creating systems that neither can control. Both have reason to prevent AI from interfering autonomously with nuclear command systems or triggering unintended military escalation.
Communication, common testing standards, emergency channels and greater transparency around genuinely catastrophic capabilities therefore deserve serious exploration. Catherine Thorbecke of Bloomberg recently argued that US-China AI talks should seek mutual visibility rather than an unrealistic grand bargain. That is sensible.
Overlaps end
But the overlap ends where strategic advantage begins.
China has little reason to help the US neutralize AI capabilities that strengthen China militarily or politically. Nor should Washington expect Beijing to disclose sensitive military, intelligence or cyber programs simply because both governments have signed an agreement bearing the words “AI safety.”
This is where global cooperation must be carefully defined. An agreement designed to prevent either country from losing control of advanced AI could benefit both.
An agreement that constrains the US’ ability to develop defensive AI while leaving unverifiable what China’s military and intelligence services are developing could make it less safe.
The most dangerous mistake would therefore be to negotiate “AI safety” as though every AI capability belonged in the same basket. They do not.
The US needs a more discriminating strategy: slow capabilities that materially increase the danger of losing human control and accelerate defensive capabilities to protect democratic societies from adversarial uses of AI. That means stronger cyber defenses.
Frightened of itself
It also means recognizing something important, which is that maintaining an AI advantage over authoritarian competitors can itself be a safety objective.
This is where calls for slowing AI development become dangerously incomplete. Asking US companies to slow broadly while assuming competitors will eventually recognize the wisdom of doing likewise is not a strategy. Authoritarian governments have strategic objectives that do not disappear because Silicon Valley becomes frightened by what it is building.
None of this means Amodei is wrong. Humanity should not race blindly toward technologies whose consequences we do not understand merely because everyone fears somebody else will arrive first.
But neither should democracies become so preoccupied with the possibility of AI turning against humanity that they neglect the more immediate possibility of humans turning AI against one another. That distinction should guide both domestic policy and negotiations with China.
The AI race is dangerous. But there is more than one way to lose it.
Racing too quickly toward capabilities we cannot control could endanger humanity. Racing too slowly toward capabilities democracies need to defend themselves could hand an extraordinary advantage to regimes that do not share their values.
The challenge, then, is not simply to put the brakes on AI. It is knowing where to brake — and when we may need to press the accelerator.
John Cheng is a retired businessman from Hong Kong who lives in Taiwan and is the author of “Taiwan Is Taiwan.”
