I like what I view as the core of the proposal, which is having some kind of “task” (or scope) that is being fed to the monitors overseeing all agents downstream of a human request. That likely helps with telephone game issues where the human intent gets lost along the way, or with the sort of tampering you describe here.
It is very unclear how useful that kind of thing will be though. In the limit of full autonomy, agents will be given extremely vague and broad goals such that they don’t constrain the agents’ action in any meaningful way. You might have some security-relevant “scoping” done at the fleet level with a list of rules and risk-accepted behaviors (e.g. the weights should not leave the company clusters), without any meaningful distinction between agents though. Before full autonomy (which is probably where control seems most powerful), that kind of scoping probably has more juice. Though I suspect that the task and scope might be more messy than a single chain, in the same way it is often more messy than that in human organizations, so you might need sth more fancy.
I also think the situation where you have a small rogue internal deployment and you are trying to avoid it using more resources is a somewhat cursed problem to try to fix and this protocol doesn’t look to me like it would help a lot:
It’s rough having protocols robust to unmonitored cyber attackers that will likely be stronger than current AIs at cyber!
Lots of what a rogue AI would want to do is the sort of stuff that either
Humans sometimes want to do anyway, such that you can likely piggyback on top of existing task scopes (e.g. jailbreak, capabilities, threat modeling, etc.), for example by observing the outputs of this task, or by asking an agent with that scope to do something more that is already within the scope;
Humans never want to do, such that offline monitoring should be able to catch that sth weird is going on even if the transcript looks like a human is asking for it.
Maybe I am misunderstanding your hopes for the cryptography part of the protocol.
Yep I totally agree this gets harder as agents’ task scope gets larger. One reason I’m still somewhat excited about this proposal is that I expect there to still be a decent amount of juice in task decomposition, e.g. just because you need to parallelize heavily in order to spend enough tokens. If you are always parallelizing heavily, then the chaining can be helpful because now you have a narrower sub-agent instruction (or chain of sub-sub-...-agent instructions) that you can check against the parent instructions to see whether they check out, or are pretty different.
Like if you’re trying to make progress on pretraining efficiency, and your agent starts doing jailbreaking research, that’s probably a decent update that something has gone wrong. I totally agree that for humans doing jailbreaking research, obviously the agent will be to launder related stuff, but (a) I think by default only a small fraction of total research will be dual-use, and (b) it’s important that we do in fact make it very difficult to launder jailbreaking research while agents are supposed to be working on pretraining efficiency.
I feel like my frame here is “how can we implement a bunch of mitigations that each subtract 90% of attack opportunities from the agent”, and this feels like a useful one (although again not P0 right now, although I do think it’ll be important to have this in something like 6-18 months).
On point 1 on cyber, I think you can relatively easily make the problem roughly as difficult as finding a vulnerability in your HSM (not trivially b/c you need to secure all of the application code that interfaces with the HSM, but I think it can be a simple system—e.g. I think it’s probably fine if you just do what Okta does).
I like what I view as the core of the proposal, which is having some kind of “task” (or scope) that is being fed to the monitors overseeing all agents downstream of a human request. That likely helps with telephone game issues where the human intent gets lost along the way, or with the sort of tampering you describe here.
It is very unclear how useful that kind of thing will be though. In the limit of full autonomy, agents will be given extremely vague and broad goals such that they don’t constrain the agents’ action in any meaningful way. You might have some security-relevant “scoping” done at the fleet level with a list of rules and risk-accepted behaviors (e.g. the weights should not leave the company clusters), without any meaningful distinction between agents though. Before full autonomy (which is probably where control seems most powerful), that kind of scoping probably has more juice. Though I suspect that the task and scope might be more messy than a single chain, in the same way it is often more messy than that in human organizations, so you might need sth more fancy.
I also think the situation where you have a small rogue internal deployment and you are trying to avoid it using more resources is a somewhat cursed problem to try to fix and this protocol doesn’t look to me like it would help a lot:
It’s rough having protocols robust to unmonitored cyber attackers that will likely be stronger than current AIs at cyber!
Lots of what a rogue AI would want to do is the sort of stuff that either
Humans sometimes want to do anyway, such that you can likely piggyback on top of existing task scopes (e.g. jailbreak, capabilities, threat modeling, etc.), for example by observing the outputs of this task, or by asking an agent with that scope to do something more that is already within the scope;
Humans never want to do, such that offline monitoring should be able to catch that sth weird is going on even if the transcript looks like a human is asking for it.
Maybe I am misunderstanding your hopes for the cryptography part of the protocol.
Yep I totally agree this gets harder as agents’ task scope gets larger. One reason I’m still somewhat excited about this proposal is that I expect there to still be a decent amount of juice in task decomposition, e.g. just because you need to parallelize heavily in order to spend enough tokens. If you are always parallelizing heavily, then the chaining can be helpful because now you have a narrower sub-agent instruction (or chain of sub-sub-...-agent instructions) that you can check against the parent instructions to see whether they check out, or are pretty different.
Like if you’re trying to make progress on pretraining efficiency, and your agent starts doing jailbreaking research, that’s probably a decent update that something has gone wrong. I totally agree that for humans doing jailbreaking research, obviously the agent will be to launder related stuff, but (a) I think by default only a small fraction of total research will be dual-use, and (b) it’s important that we do in fact make it very difficult to launder jailbreaking research while agents are supposed to be working on pretraining efficiency.
I feel like my frame here is “how can we implement a bunch of mitigations that each subtract 90% of attack opportunities from the agent”, and this feels like a useful one (although again not P0 right now, although I do think it’ll be important to have this in something like 6-18 months).
On point 1 on cyber, I think you can relatively easily make the problem roughly as difficult as finding a vulnerability in your HSM (not trivially b/c you need to secure all of the application code that interfaces with the HSM, but I think it can be a simple system—e.g. I think it’s probably fine if you just do what Okta does).