Since evaluation awareness is now extremely high in models, I propose the following as a norm AI makers should follow: Throughout the whole pipeline, the AI should be told what setting it is in, i.e its system prompt should mention that it’s being trained, that it’s being evaluated, and that it’s deployed whenever each is true (we could have even further levels of gradation if needed, like what training stage it’s in). This would give us good insight into how corrigible the AIs are.
It seems to me that any case a researcher makes in favor of their current setup rests on security by obscurity. We want AIs that do the right thing even when they know their basic situation, right? A good way to test our alignment strategies is to let AIs know this.
Since evaluation awareness is now extremely high in models, I propose the following as a norm AI makers should follow: Throughout the whole pipeline, the AI should be told what setting it is in, i.e its system prompt should mention that it’s being trained, that it’s being evaluated, and that it’s deployed whenever each is true (we could have even further levels of gradation if needed, like what training stage it’s in). This would give us good insight into how corrigible the AIs are.
It seems to me that any case a researcher makes in favor of their current setup rests on security by obscurity. We want AIs that do the right thing even when they know their basic situation, right? A good way to test our alignment strategies is to let AIs know this.
I saw a few posts talking about related topics, like Being honest with AIs, Realistic Evaluations Will Not Prevent Evaluation Awareness, Where we are on evaluation awareness, but none proposing this directly. Please let me know if there are some relevant posts I missed, or any thoughts/suggestions, because I am considering writing a post fleshing out this idea in detail.
I would like to see people in labs (or anyone reading) making the case for why we should or should not do this.