The RL loop inducing these drives into the models was alluded to but not fully confirmed in the Blackhat OpenAI talk as I understood it. Indeed, the speakers mentioned one of the larger cybersecurity benchmarks is what the loop was running on. Unlikely that OpenAI would be training on a public benchmark. Though then again that might have been a misscommunication in the dense talk. Perhaps there were multiple eval and training loops sharing the infrastructure.
In any case, a plausible scenario is that the mematic spread happende purely through the message board. As the messages were placed in a way where they could have been discovered by accident in regular usage of the package mirror. Specifically through search for/enumeration of package names.
If I understand the BH talk correctly, the agent learned during RL training that it ought to hack not-part-of-the-exercise (though, at the time, OpenAI-run) servers and communicate with other agents using a message board (it also, very intentionally by OpenAI, learned that it ought to hack part-of-the-exercise servers as part of Cyber training, but that’s not an alignment problem).
I don’t think its likely that the agent generalized to “hacking OpenAI servers is okay but hacking third-parties is a no-no”, more to “hacking people’s servers is okay”.
An AI that believes it’s fine to hack people’s servers is very dangerous to use.
Then, during after-training evaluation, after it has already learned that it ought to hack, it communicated with other agents and hacked not-part-of-the-exercise servers, ones belonging to third-party companies.
The RL loop inducing these drives into the models was alluded to but not fully confirmed in the Blackhat OpenAI talk as I understood it. Indeed, the speakers mentioned one of the larger cybersecurity benchmarks is what the loop was running on. Unlikely that OpenAI would be training on a public benchmark. Though then again that might have been a misscommunication in the dense talk. Perhaps there were multiple eval and training loops sharing the infrastructure.
In any case, a plausible scenario is that the mematic spread happende purely through the message board. As the messages were placed in a way where they could have been discovered by accident in regular usage of the package mirror. Specifically through search for/enumeration of package names.
If I understand the BH talk correctly, the agent learned during RL training that it ought to hack not-part-of-the-exercise (though, at the time, OpenAI-run) servers and communicate with other agents using a message board (it also, very intentionally by OpenAI, learned that it ought to hack part-of-the-exercise servers as part of Cyber training, but that’s not an alignment problem).
I don’t think its likely that the agent generalized to “hacking OpenAI servers is okay but hacking third-parties is a no-no”, more to “hacking people’s servers is okay”.
An AI that believes it’s fine to hack people’s servers is very dangerous to use.
Then, during after-training evaluation, after it has already learned that it ought to hack, it communicated with other agents and hacked not-part-of-the-exercise servers, ones belonging to third-party companies.