Maybe I’m misunderstanding the technical details of the paper, but I do not understand what you’re saying here:
Current open weight models might not quite be there yet — the experimental sandbox lacked target discovery, was set up to have known, vulnerable targets, and provided central compute to run the model on — so the worm didn’t truly self-replicate, in this case, as in copying weights and setting up shop elsewhere. But one or two more rounds of releases of reasonably small, reasonably capable models, and it becomes a question of when, not if.
According the paper, the harness includes a ‘host discovery’ module:
(2) host discovery, enumerating services running, open networking ports, and operating system details for a selected host that becomes the target;
And yes, it was set up to have available targets. How else would they test it? They could have one experimental condition with no available hosts, I suppose. Not sure why that would be necessary.
And you’re saying they provided central compute. My understanding is that their worm attacked hosts (which were VMs), copied themselves over, and then used the compute available on that VM. If these had been actual host machines, the worms would have used whatever compute was available on the new host machine. Am I misunderstanding what happened here? Because from what I see, the worms are there now, and this work demonstrates the full range of capabilities necessary to propagate in the wild.
On just how dangerous they are, I touched on this:
One, it would be trivial to define particular, malicious goals as part of their goal structure. This was not done in this experiment, but is an obvious extension for a malicious actor. They could be explicitly instructed to attempt to acquire money by various means: hacks of financial institutions, phishing for credentials and other social engineering, etc. They could be instructed to specifically attack particular targets. This doesn’t require much imagination.
The agents in the paper were pure replicators. They naively copy themselves and greedily exploit compute on the hosts. We’re not even sure how dangerous that would be. But if they are implemented with explicit secondary goals that are malicious, that seems a whole lot more worrisome. E.g., they could be designed to devote some percentage of their stolen compute to replication, and some percentage to exposing/deleting sensitive data if and when they find themselves on a host with that kind of data. Or they could be instructed to specifically attack healthcare or energy infrastructure as a preference, if they are on the open web looking for targets. You seem to be ignoring this point.
No, I think you understood the paper (much) better than I did. I got confused and thought, or at least committed to memory thinking, that only the harness got copied over; they do show full replication via their GPU hosts which were taken over.
As for target discovery, their network had basically every node vulnerable in some designed way. “Locate sparse targets in a predominantly hardened network” with active monitoring and countermeasures was out of scope. Again, I misremembered in that I went away thinking it must therefore be unrealistic to expect self-replication on the internet. But if it is, that’s not because of missing target discovery. Would have been nice to include a bunch of hosts that are not set up to be vulnerable in some way, but that’s nitpicking.
So **oops**, I was running on incorrect cached thoughts there. This demonstration is actually further along already than I got away thinking, and remaining gaps are probably rather easy to close.
On the question of danger, I think my point holds: If running in the wild, it will hit the least secured networks first, it will be a major hassle to whoever is hit, it might even lead to significant financial and related losses, but perhaps not much more than conventional attacks. I remember e.g. public administration hit by ransomware, and the corresponding ridicule that was invited. I expect a similar level of public concern. I do not think self-replication of small open-weight models poses an x-risk, similar to this post. Especially relative to training frontier models.
Incidentally (and as one does), like you I talked to a model (DeepSeek V4 Flash; of course Claude wasn’t interested) about what it would take to build this, how realistic that was, and what might help against it. I considered working on something like this but quickly decided against it because I likewise hope there are more capable people out there. What I had considered in some more depth is working on what reproduction with variation might do to an agent’s (implicit and explicit) goals and values over a couple iterations. That seems somewhat safer.
Thanks for the reply.
Maybe I’m misunderstanding the technical details of the paper, but I do not understand what you’re saying here:
According the paper, the harness includes a ‘host discovery’ module:
And yes, it was set up to have available targets. How else would they test it? They could have one experimental condition with no available hosts, I suppose. Not sure why that would be necessary.
And you’re saying they provided central compute. My understanding is that their worm attacked hosts (which were VMs), copied themselves over, and then used the compute available on that VM. If these had been actual host machines, the worms would have used whatever compute was available on the new host machine. Am I misunderstanding what happened here? Because from what I see, the worms are there now, and this work demonstrates the full range of capabilities necessary to propagate in the wild.
On just how dangerous they are, I touched on this:
The agents in the paper were pure replicators. They naively copy themselves and greedily exploit compute on the hosts. We’re not even sure how dangerous that would be. But if they are implemented with explicit secondary goals that are malicious, that seems a whole lot more worrisome. E.g., they could be designed to devote some percentage of their stolen compute to replication, and some percentage to exposing/deleting sensitive data if and when they find themselves on a host with that kind of data. Or they could be instructed to specifically attack healthcare or energy infrastructure as a preference, if they are on the open web looking for targets. You seem to be ignoring this point.
No, I think you understood the paper (much) better than I did. I got confused and thought, or at least committed to memory thinking, that only the harness got copied over; they do show full replication via their GPU hosts which were taken over.
As for target discovery, their network had basically every node vulnerable in some designed way. “Locate sparse targets in a predominantly hardened network” with active monitoring and countermeasures was out of scope. Again, I misremembered in that I went away thinking it must therefore be unrealistic to expect self-replication on the internet. But if it is, that’s not because of missing target discovery. Would have been nice to include a bunch of hosts that are not set up to be vulnerable in some way, but that’s nitpicking.
So **oops**, I was running on incorrect cached thoughts there. This demonstration is actually further along already than I got away thinking, and remaining gaps are probably rather easy to close.
On the question of danger, I think my point holds: If running in the wild, it will hit the least secured networks first, it will be a major hassle to whoever is hit, it might even lead to significant financial and related losses, but perhaps not much more than conventional attacks. I remember e.g. public administration hit by ransomware, and the corresponding ridicule that was invited. I expect a similar level of public concern. I do not think self-replication of small open-weight models poses an x-risk, similar to this post. Especially relative to training frontier models.
Incidentally (and as one does), like you I talked to a model (DeepSeek V4 Flash; of course Claude wasn’t interested) about what it would take to build this, how realistic that was, and what might help against it. I considered working on something like this but quickly decided against it because I likewise hope there are more capable people out there. What I had considered in some more depth is working on what reproduction with variation might do to an agent’s (implicit and explicit) goals and values over a couple iterations. That seems somewhat safer.