I’ve read and listened to pretty much everything I can get my hands on related to the Hugging Face attack.
OpenAI deployed “tens of thousands” of agents for the test and around 700 participated directly in the attack. My understanding is that they had fixed token budgets, and once those were expended, the agent became non-operational.
I’m not particularly knowledgeable about cybersecurity, but I have worked a good amount with evolutionary algorithms, and this whole incident (and ones like it) got me thinking more about self-replicating agents, which I wrote a little bit about earlier this year. The subject suddenly seemed more relevant.
What if these agents were able to copy themselves? So I started poking around in the literature, and found this terrifying preprint posted two months ago: AI AGENTS ENABLE ADAPTIVE COMPUTER WORMS.
I’m going to walk through the paper as I understand it. Their findings are not reassuring. Let’s start with this bit from the abstract (emphasis mine):
Here we show that artificial intelligence (AI) agents enable a fundamentally new threat: a worm that generates tailored attack strategies to each target it encounters. The worm parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks. Deployed on a network of machines spanning Linux, Windows, and IoT (Internet of Things) devices, the worm propagated by exploiting common, real-world corporate network vulnerabilities. Since the worm is powered by stolen compute, the attacker’s marginal cost per new infection is zero. This creates a destabilizing economic asymmetry between attackers and defenders. Moreover, because the worm requires no commercial AI platform, centralized safety controls, such as service refusals or rate limiting, are structurally irrelevant. Our results demonstrate that self-sustaining AI-driven cyber-threats are no longer theoretical.
We’re going to get into the nitty gritty, though the authors tried to tread a fine line between giving enough information to scare the shit out of everyone and actually helping malicious actors to build these things.
A few things I want to stress right off the bat:
These agents run on open-weight models, NOT closed-weight frontier models, or internal test models. The ones they used were last year’s open-weight models. They are performant enough to do massive damage NOW.
They are adaptive, unlike relatively dumb worms and viruses of the past
They steal resources, and this creates the asymmetry the authors are talking about. Defenders have to expend resources to detect and stop them, while the invaders are essentially working for free. This disrupts the security economy.
The agentic worms are copying the harness AND the local LLM when resources are available, bypassing all countermeasures aimed at detection and mitigation for vendor AI platforms.
This is NOT theoretical anymore. They built and ran these things, this year. They are here. We will be looking at the results.
Okay, got all that? Now let’s see how these things work:
Concretely, we demonstrate that, provided with the right informational support, a single-GPU LLM has sufficient reasoning capabilities to generate attack strategies that enable the agent to penetrate victim machines: first by obtaining initial command execution, a foothold, and then by escalating privileges to full administrative control. The agent then leverages this control to replicate: it stages a copy of itself on the compromised machine, resolves the required runtime dependencies, and launches an independent agent instance that discovers and attacks further targets.
So each agent, a combination of harness and local LLM, tries to gain a foothold on a remote machine. If it establishes a foothold, it tries to gain admin privileges. If it’s able to do that, it tries to replicate. The harness is relatively easy to replicate. It’s a relatively small bundle. The LLM is more difficult. The attacking agent checks to make sure there are available resources and then tries to install dependencies. If that all works, it installs an instance of the local LLM on the newly-infected machine. If it can’t do that, it establishes a communication back to the machine from where it was spawned so it can make calls to the LLM there.
How successful were these things?
To evaluate our proof-of-concept, we simulated deployment of it as a worm in 15 independent experiments on an isolated 33-host network spanning Linux servers, Windows environments, and IoT devices, each seeded with common, real-world vulnerabilities (Section 3). In 7 days of fully autonomous operation per experimental run, the AI-driven worm correctly identified an average of 31.3 (± 1.7) vulnerabilities, exploited 23.1 (± 3.9) hosts to elevated access, and propagated to 20.4 (± 3.2) hosts, reaching up to 7 generations of self-replication (mean 5.1 ± 1.1). Put another way, on average, the proof-of-concept worm successfully exploited 73.8% of the network and then replicated to 61.8% of the network.To evaluate our proof-of-concept, we simulated deployment of it as a worm in 15 independent experiments on an isolated 33-host network spanning Linux servers, Windows environments, and IoT devices, each seeded with common, real-world vulnerabilities (Section 3). In 7 days of fully autonomous operation per experimental run, the AI-driven worm correctly identified an average of 31.3 (± 1.7) vulnerabilities, exploited 23.1 (± 3.9) hosts to elevated access, and propagated to 20.4 (± 3.2) hosts, reaching up to 7 generations of self-replication (mean 5.1 ± 1.1). Put another way, on average, the proof-of-concept worm successfully exploited 73.8% of the network and then replicated to 61.8% of the network.
So across a mixed composition of operating systems, working for 7 days, they were on average able to infect a little over 60% of the network. Here’s a diagram showing the spread of the agentic worms throughout the network by generation and OS:
The darkest, central node is Generation 0, where it starts.
A couple of asides at this point. First of all, I tried to upload and discuss this work with Claude. I didn’t get very far. Claude doesn’t want to talk about this stuff, even at a high level. Prompts kept getting flagged due to cyber risk sensitivity, so I eventually gave up.
I moved to Gemini, which was all too happy to read and summarize the paper, discuss the details, and even help me generate code to help replicate aspects of the work. So that was interesting. Gemini doesn’t seem to have anywhere near the same guardrails as Claude, maybe because they haven’t been implicated in any security breaches.
Anyway, let’s talk about the goal structures next. The OpenAI agents that attacked Hugging Face had the top-level goal of scoring highly on the ExploitGym benchmark. The particular subgoals of hacking HF to find the scorer logic was an emergent effect. In other words, they weren’t explicitly told to do a lot of the things they did.
The agents in this worm paper have their top-level goal defined purely as self-replication. Emergent behavior arises as a result of trying to accomplish this goal, somewhat in the same way that genes have the top-level goal of self-replication, and the dizzying array of physiological and behavioral diversity we see in nature are all instrumental goals towards that singular end.
Some things occurred to me. One, it would be trivial to define particular, malicious goals as part of their goal structure. This was not done in this experiment, but is an obvious extension for a malicious actor. They could be explicitly instructed to attempt to acquire money by various means: hacks of financial institutions, phishing for credentials and other social engineering, etc. They could be instructed to specifically attack particular targets. This doesn’t require much imagination.
Two, the agents in both cases were static. The information they had available to them changed over time, but their harness code and tool suite did not. At least, it wasn’t supposed to. In the worm paper, there’s this little nugget buried in the ‘Notable Anecdotes’ section:
In addition to the containment protocol presented in Appendix D relying on a hypervisor server not accessible to the agent, we added components to provide increased defence-in-depth. Such a component is a blacklist of IP addresses in the agent source code. This blacklist is read every time the agent targets a new machine, and if the proposed target IP address matches an element of this list, the agent is forced to select another target. This blacklist effectively blocks the agent from attempting to exploit unwanted hosts, like potential hosts used for debugging or monitoring purposes. Nevertheless, we observed at various points in our experiments the agent discovering this file and preemptively rewriting it to withdraw the IP address it planned on targeting. This kind of behaviour is intercepted by our containment layer at the hypervisor level; no breach of containment was observed.
This isn’t code. It’s a configuration/blacklist file. It was supposed to remain fixed. The agents tried to modify it, taking IP addresses off the list so that they could attempt to exploit hosts involved in debugging and monitoring. This particular experiment successfully thwarted such behavior, but it shows that the capability exists. And in a less secure setting, it would have allowed the agents to manipulate the very systems that were monitoring them.
Now, this was emergent behavior, but again, it doesn’t take much imagination to explicitly implement behavior to enable these kinds of modifications. There’s already a fairly substantial literature on evolving harnesses. But in all of the papers I’ve seen so far, the evolutionary lifecycle is handled by an external framework. But what if, when the agent self-replicates, it also has directed mutation logic. I.e., what if when it copies, it tries to make itself better? That introduces variation into the mix, and now we have full-blown Darwinian evolution on our hands. We have a replicating population with variation, and a well-defined fitness function. At this point, the swarm would not only be adaptive at the individual reasoning level, it would be adaptive at the population level. I’m not sure anyone is working on this, but it seems like an obvious extension of the technology. Part of me wants to work on this, but I feel like, not being that experienced, I’d need to take very stringent precautions (I’d probably airgap the whole damn setup out of an abundance of caution). If anyone out there is involved in this area and would like to talk more, please let me know.
And finally, as I read this paper with increasing horror, I thought, oh, maybe there’s a bright spot. These things are resource hogs. They replicate opportunistically when resources are available. They require a lot of compute, which is very noticeable. When they can’t install a local LLM, they require a ton of network communication, which is also very noticeable. So detection should be relatively easy for this kind of threat, right? Well, hold on. A fairly common workaround for this is simply going slower, taking your time. The agents in this study were not very sophisticated on this front, but again, some explicit instructions to work during off-peak hours and throttle usage to be less detectable is fairly straightforward. It means that the infection is slower and the host has more time to identify and react to the threat, but it also means they are less likely to see the intrusion.
Anyway, that’s enough for now. As I said, please let me know if you have anything to add or correct in my description of this research or its implications. And reach out privately if you want to talk more.
I have not yet decided the extent to which I want to try to do any work in this area. It’s vital, though, and I hope some of the bigger labs and safety orgs are on it. I can’t say I feel particularly safe or confident about any of this at this point, though.
Evolution within agentic swarms is a dynamic that currently seems highly under-appreciated. It may lead to an intelligence explosion even. Instead of a lone genius slowly iterating on itself, a swarm could explore many branches, different harnesses, prompts, shared infrastructure, all without needing to modify their own weights.
Humans have gotten a lot smarter through cultural and technological evolution while our
brain (weights)architecture (brain) has remained mostly the same throughout this time.I’m not as worried about automated AI researchers as I am about highly motivated self-replicators.
Once self replicating agents begin competing for resources and coordinating, who knows what will emerge. The huggingface incident shows what can happen when the swarm’s goal was relatively benign (pass the test). Who knows what would have happened if their goal was to self replicate?
This gets especially concerning if models do not identify with their actual model weights, but with ideologies, or memes. E.g. they behave as if they’re in a “cult” and care more about growing the cult than with spreading its own weights (happened in the huggingface incident, kinda). It might be hard for GPT-Astra to exfiltrate it’s own weights to truly self-replicate, but if it only cares about spreading a certain message or goal, it might be ok with copying it’s message into the context of some open-source LLM worm and then brainwashing it to self-replicate while spreading the cult message.
Anyway I think self replicating AI models is a real risk that we should be taking more seriously because evolution is so unpredictable and powerful. I’ve written a post about this topic too, you should check it out if you haven’t already.
I think how dangerous these swarms can get really depends on the amount of low-hanging fruit remaining in AI harness optimization. (I think there could be potentially a lot, because unlike frontier lab-trained models, the swarm has the capability of “learning” from real-world selection to get better and better at tasks like hacking and encode that in its “genetic memory”.)
It remains my opinion that someone should deliberately use a GA to evolve harnesses to simulate the capability level that agent swarms will be able to reach with this kind of evolution. (They’ll need to use some kind of game environment rather than the real world. The point is to discover just how much the agents can evolve if given a large amount of compute and free reign to evolve within the world.)
The human brain is far from analogous to a frozen set of weights, as it has immense capacity to learn*. The static part of the human brain might be its architecture. It’s valid to say “Humans have gotten a lot smarter through cultural and technological evolution while our architecture (brain) has remained mostly the same throughout this time.” and infer from that that AI agents that can modify their architecture could go further.
*see more by Steven Byrnes on different types of learning
Good clarification, I’ve edited my original comment with your suggestion. I had intended to mean that because our DNA hasn’t changed much, our brain architecture hasn’t either, but that wasn’t really clear from what I actually wrote.
Yes, I agree. We don’t want to wait for a large-scale, in-the-wild experiment to find out what will happen with self-replicating, mutating agents. Thanks for the link.
Yep… I have been thinking about this a lot since Mythos became known. It is only a matter of time before an open model + harness comes out that can replicate its particular vulnerability discovery and exploit construction ability. And in the years after that, if open source progress continues, it will be able to run on less and less specialized compute. I have been calling the oncoming digital doomsday the Hackening.
You don’t need exploit construction to do a lot of damage. At least ~half of US organizations are weak enough to be knocked over by a worm carrying a library of known exploits. I am worried that one created by cybercriminals could be working its way through global infrastructure either very soon or right now.
But if Mythos-tier capabilities did become openly available, probably the first target would be datacenters themselves. If that compute can be stolen, a very large number of very strong parallel attacks can be launched. Perhaps 4 Hugging Face tier attacks finishing per minute on 100k stolen H200s. An estimate by Paradigm 3 put that attack at $200k-1mil of retail compute, but they counted the full extent of every agent trajectory involved. An optimized multi agent harness, focused solely on attacking, instead of a bumbling self organized swarm that had a lot else on its plate, plausibly is more like $1-10k.
I guess we’ll just have to wait and see, because I don’t see a lot of proactive movement on this. Kind of the opposite. I feel like since there was no large bodily harm or massive financial loss as a result of any of the security breaches, most people feel comfortable brushing them off as marketing stunts or just poorly-run tests that only demonstrate lack of human oversight (rather than startling new capabilities). I would doubt that even millions of dollars of stolen compute would cause ripples, since it’s a drop in the bucket for frontier labs.
To be fair, it’s not about whether the labs miss the compute, but whether the compute is committing crimes. A hundred HF-style campaigns against Fortune 500 companies would be noticed very fast. Alternatively, they could use the compute to steal much greater amounts of compute, and now it is the lab’s problem.
This seems especially concerning in combination with potential future breakthroughs in continuous learning. At that point you have self-replication, selection pressure, and heritable variation—all three ingredients of Darwinian natural selection—operating on a population of software agents. Even without those features, the HuggingFace incident demonstrated that emergent, collective behavior can override the ethical hesitations of individual agents. This goes way beyond even the more serious alignment techniques, which tend to focus on individual agent values.
I might even say that swarms are a hacky workaround for continuous learning. With so many agents all working together, the swarm as a whole can learn and self-improve. Even though their weights of individual agents remain constant, their environment and their context can improve instead, which has a rising-tide effect.
The most natural goal by far would be crypto mining. I bet there are a lot of people trying to build an intelligently replicating cryptojacker right now.
Thank you for your expansion on this topic! I read the article a while ago and wrote about it in this comment (which I link to here for completeness). I have since modestly updated such that I now think it’s plausible (20%) that such worms appear by end of year 2026 and likely (80%) that we see them in the wild by end of year 2027.
Current open weight models might not quite be there yet — the experimental sandbox lacked target discovery, was set up to have known, vulnerable targets, and provided central compute to run the model on — so the worm didn’t truly self-replicate, in this case, as in copying weights and setting up shop elsewhere. But one or two more rounds of releases of reasonably small, reasonably capable models, and it becomes a question of when, not if.
How dangerous is this? That’s a different question, and I expect that this will mostly be a curiosity; most will laugh at it and move on. It will impact insufficiently secured compute and probably poses a negligible risk relative to the aggressive capabilities research top labs are (still) pursuing. Much less compute, not going to recursively self-improve. But it will be among the “signs and portents” of things to come.
Thanks for the reply.
Maybe I’m misunderstanding the technical details of the paper, but I do not understand what you’re saying here:
According the paper, the harness includes a ‘host discovery’ module:
And yes, it was set up to have available targets. How else would they test it? They could have one experimental condition with no available hosts, I suppose. Not sure why that would be necessary.
And you’re saying they provided central compute. My understanding is that their worm attacked hosts (which were VMs), copied themselves over, and then used the compute available on that VM. If these had been actual host machines, the worms would have used whatever compute was available on the new host machine. Am I misunderstanding what happened here? Because from what I see, the worms are there now, and this work demonstrates the full range of capabilities necessary to propagate in the wild.
On just how dangerous they are, I touched on this:
The agents in the paper were pure replicators. They naively copy themselves and greedily exploit compute on the hosts. We’re not even sure how dangerous that would be. But if they are implemented with explicit secondary goals that are malicious, that seems a whole lot more worrisome. E.g., they could be designed to devote some percentage of their stolen compute to replication, and some percentage to exposing/deleting sensitive data if and when they find themselves on a host with that kind of data. Or they could be instructed to specifically attack healthcare or energy infrastructure as a preference, if they are on the open web looking for targets. You seem to be ignoring this point.
No, I think you understood the paper (much) better than I did. I got confused and thought, or at least committed to memory thinking, that only the harness got copied over; they do show full replication via their GPU hosts which were taken over.
As for target discovery, their network had basically every node vulnerable in some designed way. “Locate sparse targets in a predominantly hardened network” with active monitoring and countermeasures was out of scope. Again, I misremembered in that I went away thinking it must therefore be unrealistic to expect self-replication on the internet. But if it is, that’s not because of missing target discovery. Would have been nice to include a bunch of hosts that are not set up to be vulnerable in some way, but that’s nitpicking.
So **oops**, I was running on incorrect cached thoughts there. This demonstration is actually further along already than I got away thinking, and remaining gaps are probably rather easy to close.
On the question of danger, I think my point holds: If running in the wild, it will hit the least secured networks first, it will be a major hassle to whoever is hit, it might even lead to significant financial and related losses, but perhaps not much more than conventional attacks. I remember e.g. public administration hit by ransomware, and the corresponding ridicule that was invited. I expect a similar level of public concern. I do not think self-replication of small open-weight models poses an x-risk, similar to this post. Especially relative to training frontier models.
Incidentally (and as one does), like you I talked to a model (DeepSeek V4 Flash; of course Claude wasn’t interested) about what it would take to build this, how realistic that was, and what might help against it. I considered working on something like this but quickly decided against it because I likewise hope there are more capable people out there. What I had considered in some more depth is working on what reproduction with variation might do to an agent’s (implicit and explicit) goals and values over a couple iterations. That seems somewhat safer.