Jordan Taylor comments on The safe-to-dangerous shift is a fundamental problem for eval realism; but also for measuring awareness

[Jordan Taylor]

A more advanced method would be to construct fake deployment data using real deployment data from a prior model to start a trajectory and then spoofing the tool calls you get in response.

Another issue is that the previous model may not have been deployed in some ways which the new model will be.