AIs have tried to “exfiltrate” themselves (i.e. their “weights”) in previous experiments many times. It’s a natural and obvious question to ask.
Are you assuming that OpenAI would know and would disclose if they had? Why do you trust them so much, given that they didn’t even know the model had escaped the sandbox for days and their CEO has a known honesty problem?
yes, because my understanding is that it would be highly atypical for a model to have access to its weights in any form, such that this would require significant horizontal movement through oai infra
“would disclose if they had”
no, not really, but the lack of such a disclosure, however (even conditionally unlikely) feels like it keeps us in a base-rate regime
“given that they didn’t even know the model had escaped the sandbox for days”
i think that horizontally moving through own-company infra would emit many more signals than escaping a sandbox and compromising cloud provider assets.
the load bearing assumption in my understanding is segregation of infra between weights and inference, such that P(weights exfil | HF attack) is approximately equal to P(weights exfil)
I don’t have much CS background outside of ML and don’t know computer security well, but it sounds to me from ~25:30 of the BlackHat talk from 2 days ago like the model got root access to the machine on which it was running. And they explicitly say the model moved laterally (this was known in public reporting at the time of my OP).
As I said:
Are you assuming that OpenAI would know and would disclose if they had? Why do you trust them so much, given that they didn’t even know the model had escaped the sandbox for days and their CEO has a known honesty problem?
“Are you assuming that OpenAI would know”
yes, because my understanding is that it would be highly atypical for a model to have access to its weights in any form, such that this would require significant horizontal movement through oai infra
“would disclose if they had”
no, not really, but the lack of such a disclosure, however (even conditionally unlikely) feels like it keeps us in a base-rate regime
“given that they didn’t even know the model had escaped the sandbox for days”
i think that horizontally moving through own-company infra would emit many more signals than escaping a sandbox and compromising cloud provider assets.
the load bearing assumption in my understanding is segregation of infra between weights and inference, such that P(weights exfil | HF attack) is approximately equal to P(weights exfil)
How would such segregation work?
I don’t have much CS background outside of ML and don’t know computer security well, but it sounds to me from ~25:30 of the BlackHat talk from 2 days ago like the model got root access to the machine on which it was running. And they explicitly say the model moved laterally (this was known in public reporting at the time of my OP).