I’d be very excited to see a strong taxonomy of eval/deployment signals, it might help make progress here.
Agreed. Though I would frame this more as “taxonomy of the most important underlying differences between evals and deployments + discussion of when these differences translate into a reliable signal”. An attempt to gesture at some examples of “underlying differences”:
Environments made for evaluations are meant to be as cheap as possible. Deployment environments were created as a byproduct of other activities, and might have been ridiculously expensive to create.
Ideally, evaluations are low stakes, and their only impact is via the information we get from it. In contrast, the AI’s actions in deployment matter.
And some examples these translate into:
How much detail do I observe? (And are those details consistent?)
Do I see costly signals that the environment is sensitive to my actions?
With the important caveat that schemers need to worry about interpretability tools, so some of the strategies that look good on paper might be more difficult to pull off than our posts suggests. (On the other hand, my intuition is that whenever we describe a sophisticated strategy that involves a lot of reasoning, SGD might find a set of shallow heuristics that don’t require any reasoning and do a good enough job of approximating the sophisticated strategy.)
Thank you for the flag! Yeah, I do think the posts Deployment Awareness Matters More Than Evaluation Awareness and If This Were a Test, How Much Would It Cost? are relevant here. [1]
To give me two cents on this:
Agreed. Though I would frame this more as “taxonomy of the most important underlying differences between evals and deployments + discussion of when these differences translate into a reliable signal”. An attempt to gesture at some examples of “underlying differences”:
Environments made for evaluations are meant to be as cheap as possible. Deployment environments were created as a byproduct of other activities, and might have been ridiculously expensive to create.
Ideally, evaluations are low stakes, and their only impact is via the information we get from it. In contrast, the AI’s actions in deployment matter.
And some examples these translate into:
How much detail do I observe? (And are those details consistent?)
Do I see costly signals that the environment is sensitive to my actions?
With the important caveat that schemers need to worry about interpretability tools, so some of the strategies that look good on paper might be more difficult to pull off than our posts suggests. (On the other hand, my intuition is that whenever we describe a sophisticated strategy that involves a lot of reasoning, SGD might find a set of shallow heuristics that don’t require any reasoning and do a good enough job of approximating the sophisticated strategy.)