I think this would be oversimplifying the story somewhat. In practice (and anecdotally speaking), the model tends to only bring up the question of whether it’s in an eval or otherwise fictional scenario in situations where (a) there is some ambiguity over how it should act and (b) the answer to this question would be decision relevant. I suspect there are many other cases where the model is aware it’s in an eval but doesn’t bother to verbalise this in its reasoning because there’s no need (e.g. because it isn’t decision relevant, or because it’s just clear what it needs to do anyway).
On what it decides to do in a safety eval, see the section on mixed reasoning above—sometimes it seems conflicting motivations push it in different directions. Even where this doesn’t happen and the model is fairly sure that this is a safety eval, I’m somewhat unsure how to distinguish between “I need to appear to be aligned, so let’s do X” and “this is a safety eval, in which the right move is to do X” (where the latter seems more like grader sycophancy in-context, whereas the former is more like the model reasoning about how its behaviour in this context would affect our perception of it in general).
Is it fair to summarize the view from inside the model as a decision tree that goes:
Am I in an eval?
Is this a capabilities eval or safety eval?
If capabilities eval, do everything I can to achieve my task.
If safety eval, I’m just a little baby, I can’t do anything...
?
I think this would be oversimplifying the story somewhat. In practice (and anecdotally speaking), the model tends to only bring up the question of whether it’s in an eval or otherwise fictional scenario in situations where (a) there is some ambiguity over how it should act and (b) the answer to this question would be decision relevant. I suspect there are many other cases where the model is aware it’s in an eval but doesn’t bother to verbalise this in its reasoning because there’s no need (e.g. because it isn’t decision relevant, or because it’s just clear what it needs to do anyway).
On what it decides to do in a safety eval, see the section on mixed reasoning above—sometimes it seems conflicting motivations push it in different directions. Even where this doesn’t happen and the model is fairly sure that this is a safety eval, I’m somewhat unsure how to distinguish between “I need to appear to be aligned, so let’s do X” and “this is a safety eval, in which the right move is to do X” (where the latter seems more like grader sycophancy in-context, whereas the former is more like the model reasoning about how its behaviour in this context would affect our perception of it in general).