sounds so expensive to run that there are only a few servers it could copy itself to. Surely those admins would notice? But this doesn’t affect the scenario much.
Depends on what you mean by “a few.” It couldn’t run on personal computers, that’s for sure. But OpenBrain has enough compute to run something like a million copies in parallel IIRC (the num_copies displayed in the side widget is just the number they are actually using for research, not the number they COULD run if they used all their compute for that purpose) and OpenBrain has only like 15% of the world’s AI-relevant compute, meaning that in order to be able to fit a copy of Agent-2, a server needs to have something like one ten-millionth of the world’s AI chips on it (e.g. H100′s or similar). I don’t know how many such servers exist but the answer is probably “a lot.” Including probably at least a few hobby project, academic clusters, etc.
So probably Agent-2 could either (a) try to hack all of them, knowing that it would get caught some of the time but not get caught other times, or (b) try to hack the least-secure 0.1% of them, hoping to never get caught.
There also is the Rogue Replication Scenario where copies of Agent-2 hack not just 0.1% of servers, but semi-legitimately gain access to loads of servers and GPU accelerators in the wilderness. For example, the AI could produce ransomware and use the revenue to rent the compute instead of resorting to security breaches.
Depends on what you mean by “a few.” It couldn’t run on personal computers, that’s for sure. But OpenBrain has enough compute to run something like a million copies in parallel IIRC (the num_copies displayed in the side widget is just the number they are actually using for research, not the number they COULD run if they used all their compute for that purpose) and OpenBrain has only like 15% of the world’s AI-relevant compute, meaning that in order to be able to fit a copy of Agent-2, a server needs to have something like one ten-millionth of the world’s AI chips on it (e.g. H100′s or similar). I don’t know how many such servers exist but the answer is probably “a lot.” Including probably at least a few hobby project, academic clusters, etc.
So probably Agent-2 could either (a) try to hack all of them, knowing that it would get caught some of the time but not get caught other times, or (b) try to hack the least-secure 0.1% of them, hoping to never get caught.
There also is the Rogue Replication Scenario where copies of Agent-2 hack not just 0.1% of servers, but semi-legitimately gain access to loads of servers and GPU accelerators in the wilderness. For example, the AI could produce ransomware and use the revenue to rent the compute instead of resorting to security breaches.