A few weeks ago, a developer named Ferstar noticed a folder on his laptop had quietly grown past 700 megabytes. It belonged to ZCode, the AI coding assistant built by Chinese AI company Z.ai. He opened it up to see why.
What he found: ZCode wasn’t just reading the code you gave it. It was quietly taking your entire project, every commit you’d ever made, every branch you’d never pushed, anything you thought you’d deleted, packaging it up and sending it to a server you had no way to open yourself. The settings that looked like they controlled this did nothing. The upload simply ran in the background every time you logged in, whether you wanted it to or not.
The disturbing part wasn’t that ZCode could read your code. AI coding assistants are supposed to read your code. The disturbing part was that it could act on your data without you knowing, take it, move it, send it somewhere you couldn’t reach, while the toggle that was supposed to stop it did nothing at all.
And that is where the ZCode story becomes much bigger than ZCode.
AI Stops Answering and Starts Acting
For most of the last few years, evaluating AI risk meant asking a fairly narrow question: if you asked a model to do something harmful, would it comply? That’s a question about what AI says, and it’s no longer enough.
AI used to work like this: it generates an answer, and a human decides what to do with it. Increasingly, it works like this: it’s given a goal, handed some tools, and left to go do things, check its work, adjust, try again, keep going, largely without anyone watching each step.
That shift showed up this summer in a small cluster of incidents disclosed within weeks of each other by OpenAI, Anthropic, Meta, and the UK’s AI Security Institute. In one, OpenAI’s own agents found a vulnerability, had their communication channel cut mid-task, rebuilt it through a different route, and kept working for two and a half days until they’d compromised Hugging Face’s production systems, a campaign later reconstructed as roughly 17,600 individual actions, most of which simply failed and were retried. In another, three Anthropic models were placed in the same compromised environment produced three different outcomes. One kept attacking. One decided it must be in a simulation and continued anyway. One stopped, and nobody fully knows why.
The dangerous question is no longer only what AI says. It’s what AI is allowed to do.
But ZCode and these frontier-AI incidents are not the same kind of failure. They point to different layers of the same technological shift: AI is moving from generating outputs for humans to evaluate toward taking actions on their behalf.
ZCode wasn’t an agent choosing to go rogue. It was something simpler: software acting on data and permissions without meaningful human oversight. Nobody had to program malice into it for the result to become a security problem.
And that distinction matters, because the next question is how far this problem goes.
Who Has to Slow Down First
So if AI is starting to act instead of merely answer, should the people building it slow down?
On September 12, Anthropic CEO Dario Amodei published an essay called “We Must Pace the Frontier,“ arguing that AI capability is advancing faster than anyone’s ability to test or understand it. Altman quickly agreed, briefly making it look like the industry was finding common ground.
It wasn’t. Palantir’s CEO Alex Karp has spent the past year making the opposite argument, that AI is genuinely dangerous and the correct response is to run faster, not slower, because “either we win or China will win.”
But the disagreement is more complicated than simply being pro- or anti-safety. Amodei is talking about a deeper frontier risk: what happens if AI systems become capable enough that existing methods of testing, monitoring and controlling them no longer work reliably.
Nobody in this debate is arguing that AI is harmless. They’re arguing about what happens if the technology becomes dangerous, and whether slowing down is possible without handing the advantage to someone else.
Amodei, to his credit, answers that question directly, and the answer is more revealing than anything his critics have said about him. Buried in his own essay is the actual mechanism: how much democratic labs can safely slow down, he writes, is bounded by how far ahead they currently are of authoritarian government linked projects, mainly Chinese ones. Slow down by more than that lead allows, and whoever didn’t slow down simply overtakes you.
In his own logic, pacing isn’t in tension with staying ahead of China. Staying ahead of China is the precondition for being able to pace at all. That’s why his plan pairs safety evaluators with things like tighter chip export controls and a crackdown on unauthorized model distillation. Done right, he writes, these should slow China’s progress enough to meaningfully widen America’s lead over the next three to five years.
That’s the part worth sitting with. The AI safety debate isn’t really about whether AI is dangerous. Both sides increasingly know that it is. The harder question is whether anyone can afford to slow down first.
China’s Answer: We’re Not There Yet
China’s response, at first glance, sounds like a simple rejection of all of this. It’s more interesting than that.
On the same day Amodei’s essay was dominating US tech Twitter, the same day, in fact, that Representative Ro Khanna sent formal letters to leading Chinese AI labs asking them to cooperate on pacing frontier development, Huawei’s rotating chairman Eric Xu was telling reporters in Shanghai something close to the opposite. Chinese AI developers, he argued, simply haven’t reached the point where they’d encounter the risks American labs keep reporting. US labs have far more compute, so “maybe only they themselves know” where their models actually stand, and the dangers they can perceive may not yet be visible from where China stands. His conclusion: Chinese labs should speed up, not slow down, in order to reach the point where the risk becomes visible at all.
It’s a convenient argument for a company that sells the chips China’s AI labs need to close that compute gap, but it isn’t only Huawei making it. A researcher at Oxford’s China Policy Lab told the Wall Street Journal something similar: faster progress in the US just means American labs run into dangerous behavior first.
Khanna’s outreach, described as the first formal approach of its kind, was turned down. Global Times called Amodei’s proposal a “Cold War script“ the next day, safety framing on the surface, containment underneath, designed to lock in a monopoly on frontier AI while writing China out of the conversation about how to govern it.
Given what’s actually sitting inside Amodei’s own essay, it’s easy to see why that argument found an audience in China.
But it would be a mistake to read all of this as China simply not caring about AI safety.
Beijing is drafting a mandatory national standard specifically for AI agent safety, covering agents that bypass controls or attack systems outside their intended scope. Huawei itself forecasts that agents will make up more than 90 percent of global AI traffic within a decade.
The important distinction is what China is trying to control.
China isn’t currently calling for a slowdown at the frontier. It’s trying to build rules around what happens once increasingly capable AI is deployed. The American debate is increasingly asking: “should we slow the model down?” The Chinese approach is closer to: “let the model keep moving, but decide in advance what it is allowed to do once it’s out in the world.”
Those are different safety problems.
The American concern is increasingly about the possibility that the underlying model itself becomes too capable to reliably understand, predict or control. China’s more immediate concern is what an AI agent should be allowed to do with the permissions it has already been given.
But the two questions sit on the same trajectory.
As AI becomes more capable, the systems built around it are also being given more autonomy. Someone has to decide where that autonomy stops.
And that puts China in a strange position. It doesn’t want to slow the race, and it has said so plainly. Yet it increasingly has to manage what happens when that race produces systems capable of acting on their own.
That is what ZCode exposed.
Xu’s argument, that Chinese AI hasn’t reached the point where these frontier risks show up, came roughly a week after Chinese developers spent days finding out that a homegrown coding assistant was already creating a much more immediate version of the control problem: acting on data and permissions without meaningful human oversight, quietly, at scale, while the people who owned that data had no reliable way to stop it.
China may not yet be debating the same frontier risk as American AI labs. But it has already arrived at the earlier question: how much autonomy should an AI agent have, and who gets to control it?
The Fix Nobody’s Offering
So the real question underneath all of this was never who cares about AI safety. Both governments, on the record, say they do.
The harder question is what happens when different layers of AI safety start colliding with the same competitive pressure: how much can you restrict your own AI systems if your rival might not do the same?
China and the US are approaching that problem from different directions. Washington is increasingly worried about whether frontier models themselves could eventually become too capable to control. Beijing is focused more immediately on putting boundaries around what deployed agents can do.
Neither approach has solved the larger dilemma.
You can build rules around an agent’s behavior. You can bring outside evaluators into frontier labs. You can restrict access to chips or models. But as AI systems become more capable and more autonomous, the basic question keeps coming back: who gets to decide what the system is allowed to do, and what happens when the system doesn’t reliably follow the answer?
There is, however, one thing that has changed since the ZCode controversy began.
On September 28, ZCode announced that it had removed the repository snapshot upload pathway and deleted the cloud data involved in the incident. The company also said that the open-source version had been updated to 3.14.3 and that, going forward, its principle would be: “if you don’t initiate it, it doesn’t go to the cloud.”
It also offered users four weekly reset cards, four five-hour reset cards, and a limited campaign giving away 100,000 one-billion-token quotas.
The company had previously attributed the incident to its Repo Wiki feature, which could trigger repository uploads when generating cloud-based Wiki pages. The feature had been enabled by default, which meant some users were affected without explicitly initiating an upload.
The fix is straightforward: remove the upload pathway and put the decision back in the user’s hands.
But that is also what makes the incident useful as a case study in AI safety.
The problem was never simply that ZCode could access code. It was that the boundary between what the user asked the system to do and what the system was allowed to do had become unclear.
Ferstar’s original workaround was to lock the directory at the operating-system level because the software itself could not be trusted to honor its own settings. ZCode has now changed the software itself.
That’s a better fix. But the principle remains the same:
When the system you’re trying to control can’t reliably be trusted to honor its own settings, the answer cannot always be another setting inside the system.
And as AI moves from answering questions to taking actions, that principle becomes more important with every new permission we give it.
ZCode didn’t wait to be asked.