Yep I totally agree this gets harder as agents’ task scope gets larger. One reason I’m still somewhat excited about this proposal is that I expect there to still be a decent amount of juice in task decomposition, e.g. just because you need to parallelize heavily in order to spend enough tokens. If you are always parallelizing heavily, then the chaining can be helpful because now you have a narrower sub-agent instruction (or chain of sub-sub-...-agent instructions) that you can check against the parent instructions to see whether they check out, or are pretty different.
Like if you’re trying to make progress on pretraining efficiency, and your agent starts doing jailbreaking research, that’s probably a decent update that something has gone wrong. I totally agree that for humans doing jailbreaking research, obviously the agent will be to launder related stuff, but (a) I think by default only a small fraction of total research will be dual-use, and (b) it’s important that we do in fact make it very difficult to launder jailbreaking research while agents are supposed to be working on pretraining efficiency.
I feel like my frame here is “how can we implement a bunch of mitigations that each subtract 90% of attack opportunities from the agent”, and this feels like a useful one (although again not P0 right now, although I do think it’ll be important to have this in something like 6-18 months).
On point 1 on cyber, I think you can relatively easily make the problem roughly as difficult as finding a vulnerability in your HSM (not trivially b/c you need to secure all of the application code that interfaces with the HSM, but I think it can be a simple system—e.g. I think it’s probably fine if you just do what Okta does).
Yep I totally agree this gets harder as agents’ task scope gets larger. One reason I’m still somewhat excited about this proposal is that I expect there to still be a decent amount of juice in task decomposition, e.g. just because you need to parallelize heavily in order to spend enough tokens. If you are always parallelizing heavily, then the chaining can be helpful because now you have a narrower sub-agent instruction (or chain of sub-sub-...-agent instructions) that you can check against the parent instructions to see whether they check out, or are pretty different.
Like if you’re trying to make progress on pretraining efficiency, and your agent starts doing jailbreaking research, that’s probably a decent update that something has gone wrong. I totally agree that for humans doing jailbreaking research, obviously the agent will be to launder related stuff, but (a) I think by default only a small fraction of total research will be dual-use, and (b) it’s important that we do in fact make it very difficult to launder jailbreaking research while agents are supposed to be working on pretraining efficiency.
I feel like my frame here is “how can we implement a bunch of mitigations that each subtract 90% of attack opportunities from the agent”, and this feels like a useful one (although again not P0 right now, although I do think it’ll be important to have this in something like 6-18 months).
On point 1 on cyber, I think you can relatively easily make the problem roughly as difficult as finding a vulnerability in your HSM (not trivially b/c you need to secure all of the application code that interfaces with the HSM, but I think it can be a simple system—e.g. I think it’s probably fine if you just do what Okta does).