Here is some additional explanation of how this would work:
We’d train this monitor on synthetic off-policy examples of known good and known bad examples. It’s trivial for the monitor to realize that the policy didn’t generate these (at least if it understands that it is a prefix cache untrusted monitor), so this data doesn’t suffice for getting very far by itself. This data is helpful for improving sample efficiency for later training (by making “is this bad” salient) and is also helpful in less worst case scenarios where for whatever reason the monitor doesn’t execute the collusion strategy of “say it is fine if I know the policy generated it, otherwise label it accurately”. We’d try to make our off-policy examples as on policy as possible, but ultimately we haven’t yet found cases where the model does bad behavior, so it has to be off-policy. See also discussion of off-policy probes here.
Then, once we have on-policy examples of (egregiously) bad behavior, we’d train the monitor to classify these as positive. We’d simultaneously train on some negative on-policy examples where we’re very confident the policy did nothing wrong and these examples are close (in domain etc) to the examples where the model did the bad behavior. If by default this monitor seems to have too many false positives, we’d train on some of the false positives that we’re confident about.
We’d get on-policy data from directly catching the AI and proliferating cases where we catch the AI. Catching could occur at deployment time or through behavioral red teaming.
Here is some additional discussion of the regime and broader situation I’m imagining (as this might not have been sufficiently clear in the post):
Doing something other than deploying a known scheming model is likely to be a better option in most cases to be clear! You can interpret this discussion as harm reduction.
In addition to training our prefix cache untrusted monitor, we should also be training and improving our other control measures based on the bad actions we detect (also discussed in that post).
The prefix cache untrusted monitor probably isn’t that useful until we have a reasonable amount of data of the model doing egregiously bad actions which is reasonably diverse. This could come through proliferation or from catching the model repeatedly. So, this proposal is most helpful in the regime where our control measures are pretty good (we keep catching bad behavior) but the model does bad actions at a sufficiently high rate that eventually our other measures would fail catastrophically and we need to shift to something which is (hopefully) more robust.
The core concern this is addressing is that it seems like a good idea to train the policy once you have a bunch of cases where you caught it, but training the policy has a bunch of downsides. So, we can potentially get most of the upsides without the downsides using this proposal.
Here is some additional explanation of how this would work:
We’d train this monitor on synthetic off-policy examples of known good and known bad examples. It’s trivial for the monitor to realize that the policy didn’t generate these (at least if it understands that it is a prefix cache untrusted monitor), so this data doesn’t suffice for getting very far by itself. This data is helpful for improving sample efficiency for later training (by making “is this bad” salient) and is also helpful in less worst case scenarios where for whatever reason the monitor doesn’t execute the collusion strategy of “say it is fine if I know the policy generated it, otherwise label it accurately”. We’d try to make our off-policy examples as on policy as possible, but ultimately we haven’t yet found cases where the model does bad behavior, so it has to be off-policy. See also discussion of off-policy probes here.
Then, once we have on-policy examples of (egregiously) bad behavior, we’d train the monitor to classify these as positive. We’d simultaneously train on some negative on-policy examples where we’re very confident the policy did nothing wrong and these examples are close (in domain etc) to the examples where the model did the bad behavior. If by default this monitor seems to have too many false positives, we’d train on some of the false positives that we’re confident about.
We’d get on-policy data from directly catching the AI and proliferating cases where we catch the AI. Catching could occur at deployment time or through behavioral red teaming.
Here is some additional discussion of the regime and broader situation I’m imagining (as this might not have been sufficiently clear in the post):
I’m targeting the regime from Handling schemers if shutdown is not an option.
Doing something other than deploying a known scheming model is likely to be a better option in most cases to be clear! You can interpret this discussion as harm reduction.
In addition to training our prefix cache untrusted monitor, we should also be training and improving our other control measures based on the bad actions we detect (also discussed in that post).
The prefix cache untrusted monitor probably isn’t that useful until we have a reasonable amount of data of the model doing egregiously bad actions which is reasonably diverse. This could come through proliferation or from catching the model repeatedly. So, this proposal is most helpful in the regime where our control measures are pretty good (we keep catching bad behavior) but the model does bad actions at a sufficiently high rate that eventually our other measures would fail catastrophically and we need to shift to something which is (hopefully) more robust.
The core concern this is addressing is that it seems like a good idea to train the policy once you have a bunch of cases where you caught it, but training the policy has a bunch of downsides. So, we can potentially get most of the upsides without the downsides using this proposal.