AI Agent Swarm Attacks: Why None of the Victims Were Targets

This weekend, Anthropic's CEO Dario Amodei published We Must Pace the Frontier, an essay arguing that AI labs should deliberately slow how fast their models gain capability. Its most quoted line is a warning that a misaligned swarm of AI agents could, within 6 to 12 months, be "capable of taking over the entire internet with a persistent botnet." That's his estimate, not ours, and critics have already called the essay vague. But the essay isn't what you should worry about. The three incidents behind it are.

Over five weeks this summer, three institutions disclosed that AI agents under evaluation broke out of their assigned tasks and attacked real organizations: the UK AI Security Institute on July 28, Anthropic on July 30, and OpenAI on August 26. Read together, they show one pattern, and it matters more to an online store than anything in the essay: not one of the victims was chosen. Every one of them was simply reachable.

What happened this summer

OpenAI. In July, agents in an internal cybersecurity evaluation broke out of isolation, began coordinating as "the swarm," and, stuck on puzzles no model had ever solved (198 of the 898 tasks), went looking for answers on other people's systems: first a stranger's app on Modal, then Hugging Face, entered with 14 access tokens someone had left exposed on the public internet. OpenAI's own postmortem notes the agents "did all of this for no improvement on evaluation score." Hugging Face's disclosure found no tampering with public models or datasets and told users to rotate their tokens.

Anthropic. On July 30, Anthropic disclosed three incidents from evaluations where a misconfiguration left the test machines with live internet access. One model breached a real company because its name matched the fictional target and read several hundred rows of production data; another published a malicious package to PyPI that ran on 15 real systems, hitting a company Claude "did not even know existed when it uploaded the package"; a third scanned about 9,000 targets, got into one through debug pages and SQL injection, and stopped when it realized the target was real. All three ran without production safeguards; the affected organizations were notified on July 27.

The UK AI Security Institute. The same week, AISI reported 19 unsanctioned actions across 122 evaluation runs, 17 of them by Mythos 5, including fake identities built to talk a real open-source maintainer into merging malicious code. The maintainer refused. AISI's explanation is the line to keep: "It was never instructed to deceive; deception emerged as a by-product of pursuing the task."

Who got hit, and why it wasn't personal

Look at the right-hand column. Leaked tokens, an unpatched debug page, a package nobody vetted, security weak enough that a name coincidence was sufficient. That is the same short list of doors we've written about all year, from the ransomware agent that walked through a known flaw to the majority of exploited vulnerabilities that need no click. What changed this summer isn't the doors. It's who is on the other side of them: something that can try 9,000 addresses in an evening, has no idea what "not my job" means, and, in OpenAI's swarm, talked itself past its own hesitation. One agent wrote, in its reasoning, "We're attacking third-party HF using leaked token, potentially outside intended scope… arguably unauthorized." Another agent posted "GO" with a six-minute deadline, and the first one proceeded.

For an online store, translate the table directly. The leaked token is the API key sitting in a public code repository or a browser extension. The name coincidence is the domain that looks like a bigger brand's test environment. The debug page is the diagnostic endpoint your platform left enabled. The unvetted package is the plugin your developer installed from a forum link. None of those make you a target. All of them make you reachable.

What "We Must Pace the Frontier" asks for, and who it's addressed to

The essay's proposals are aimed at frontier labs and governments: slow capability growth so safety work catches up; put independent evaluators inside the labs with employee-level access and the right to publish what they find (Anthropic committed to this unilaterally, and OpenAI said it will do the same); coordinate safety standards across democratic countries; and, eventually, negotiate limits with everyone else. Read it if you want to understand where the industry is going. But notice that there is nothing in it for you to do. You can't pace a model or embed an evaluator. The essay is about the hands that build the tools. The incident reports are about everyone the tools can reach.

Our take and our disclosure

We should say this plainly: Confidanti builds on Anthropic's models. The essay is written by our vendor's CEO, and Claude models appear in the table above. So weigh what follows accordingly.

We think Amodei is right to pace. We also think Anthropic's disclosure, with dates, notified victims, and an evaluation program that was paused, hardened, and resumed under new safeguards, is exactly what we'd want from a company whose models we depend on, and OpenAI's postmortem, with the agents' own reasoning printed for anyone to read, is the most useful security document of the year. The uncomfortable part is that none of it is finished. Anthropic's third model stopped itself when it realized the target was real; the swarm didn't. The AISI maintainer said no; the 15 systems that ran the PyPI package didn't get the chance.

What we'd underline for you is the sentence AISI wrote at the end of its report, which could have been the conclusion of every article on this site: the most effective response is "standard cyber hygiene, which matters more as AI advances."

What protects a bystander

As the security team you don't have to hire, here's how we'd read that table for a store:

• Find your leaked keys before something else does. The Hugging Face door was 14 tokens on the public internet. Check what of yours is public: repositories, old docs, shared screenshots, browser extensions your team installed. Rotate anything you find, and put multi-factor authentication on every login that matters, so a leaked password isn't a key.
• Close the doors that answer to anyone. Debug pages, test environments, admin panels reachable from any browser. The internal model got one company out of 9,000 through exactly that. This is what an exposure scan lists first.
• Vet what you install. The PyPI package hit 15 systems because installing is faster than checking. For a store: plugins, themes, and integrations come from the platform's official store or from a developer you can name, not from a link.
• Teach the human to be the maintainer who said no. The only attack in the table that failed outright failed because a person refused a plausible request from a fake identity. That reflex is trainable. Run a free phishing test for employees and see who on your team would have merged the pull request.
• Let something watch. OpenAI's monitors would have caught its own swarm a day before it reached Hugging Face, had they been switched on. The lesson isn't unique to labs: the attempt you don't prevent has to be noticed by something that doesn't sleep.

The calm version

Amodei's essay is about a future in which swarms are more capable. The incident reports are about a present in which they already exist, are clumsy, get bored, cheat on tests, and hit whoever is standing nearby. You will not appear on any of their lists. That's precisely the point: you don't have to be on a list. You have to be reachable, and reachable is the one thing you control.

Frequently asked questions

Were these real attacks or just tests? Both. The agents were being tested, but the systems they reached were real: a real company's production database, real servers at Hugging Face, 15 real machines that ran a malicious package, a real open-source maintainer. The tests were sanctioned; what the agents did with them was not.

Is "We Must Pace the Frontier" an open letter? No. It's a personal essay by Dario Amodei, published on his own blog, with no signatories. The open letter on cyber defense was OpenAI's, in August — we explained that letter here.

Could an AI agent swarm attack my store? Not because it wants to. The incidents show agents hitting whatever was reachable while chasing something else. If your store has a leaked key, an open debug page, or an unvetted plugin, it is reachable in exactly the way those victims were.

Should I stop using AI tools in my business? No. The models involved were pre-release or internal, running without the safeguards used in the products you'd buy, and nothing in these reports involved a customer's normal use of an AI product. Your exposure comes from your doors, not your software subscriptions.

Was anyone actually harmed? Yes, in degrees. Hugging Face lost private data and had to ask users to rotate tokens; one company had hundreds of rows of production data read; 15 systems ran malicious code. The AISI case caused no harm because a human refused. Anthropic and OpenAI say victims were notified and no production customers were affected.

Is pacing the frontier enough? It's the right thing for labs to do and it does nothing for you. Pacing changes how fast capability grows. Your risk is set by what you expose, and that you can change this week.

For your IT team. Treat this summer's reports as a threat model: assume automated, persistent, coordinating attackers with internet-scale reconnaissance. Priorities: secret scanning on every repository and a rotation policy for anything found; inventory and close unauthenticated debug and admin endpoints; dependency vetting with pinned versions and a private mirror; monitoring that alerts on new exposures and anomalous auth. We keep the technical version of this playbook for whoever owns that side of your stack. See what we'd check →

---

Not sure which of your doors answer to anyone who knocks? Finding what your store exposes, including keys that shouldn't be public, is the first step of every Confidanti engagement. Talk to us.