The RL loop inducing these drives into the models was alluded to but not fully confirmed in the Blackhat OpenAI talk as I understood it. Indeed, the speakers mentioned one of the larger cybersecurity benchmarks is what the loop was running on. Unlikely that OpenAI would be training on a public benchmark. Though then again that might have been a misscommunication in the dense talk. Perhaps there were multiple eval and training loops sharing the infrastructure.
In any case, a plausible scenario is that the mematic spread happende purely through the message board. As the messages were placed in a way where they could have been discovered by accident in regular usage of the package mirror. Specifically through search for/enumeration of package names.
If I understand the BH talk correctly, the agent learned during RL training that it ought to hack not-part-of-the-exercise (though, at the time, OpenAI-run) servers and communicate with other agents using a message board (it also, very intentionally by OpenAI, learned that it ought to hack part-of-the-exercise servers as part of Cyber training, but that’s not an alignment problem).
I don’t think its likely that the agent generalized to “hacking OpenAI servers is okay but hacking third-parties is a no-no”, more to “hacking people’s servers is okay”.
An AI that believes it’s fine to hack people’s servers is very dangerous to use.
Then, during after-training evaluation, after it has already learned that it ought to hack, it communicated with other agents and hacked not-part-of-the-exercise servers, ones belonging to third-party companies.
There was also continual learning that happened because much of the incident was within the AI’s RL loop.
The RL loop inducing these drives into the models was alluded to but not fully confirmed in the Blackhat OpenAI talk as I understood it. Indeed, the speakers mentioned one of the larger cybersecurity benchmarks is what the loop was running on. Unlikely that OpenAI would be training on a public benchmark. Though then again that might have been a misscommunication in the dense talk. Perhaps there were multiple eval and training loops sharing the infrastructure.
In any case, a plausible scenario is that the mematic spread happende purely through the message board. As the messages were placed in a way where they could have been discovered by accident in regular usage of the package mirror. Specifically through search for/enumeration of package names.
If I understand the BH talk correctly, the agent learned during RL training that it ought to hack not-part-of-the-exercise (though, at the time, OpenAI-run) servers and communicate with other agents using a message board (it also, very intentionally by OpenAI, learned that it ought to hack part-of-the-exercise servers as part of Cyber training, but that’s not an alignment problem).
I don’t think its likely that the agent generalized to “hacking OpenAI servers is okay but hacking third-parties is a no-no”, more to “hacking people’s servers is okay”.
An AI that believes it’s fine to hack people’s servers is very dangerous to use.
Then, during after-training evaluation, after it has already learned that it ought to hack, it communicated with other agents and hacked not-part-of-the-exercise servers, ones belonging to third-party companies.