Is there a reason why you can’t just prompt them to do this?
You definitely could—but I’m a bit suss of prompted elicitation for tasks off-distribution relative to training—so this will be a ‘looser’ lower-bound on human red-team performance (which is the upper-bound on eval awareness).
I imagine a smart enough model could check system clocks, server clocks, or possibly even program logs to get a sense of how much time has passed in-between various actions. The evaluation environment could try and spoof these sources of time info, but the model might be smart enough to detect that. Overall I think it’s probably fine to assume that how much time is passing is available to the LLMs if they wanted to know it.
Maybe—but this does sound like an empirical question about the LLMs capability and its environment and I’d prefer to measure it empirically. You could conservatively assume LLMs know the time, and then if you fail to rule out sufficient awareness, try to take a tighter bound.
You definitely could—but I’m a bit suss of prompted elicitation for tasks off-distribution relative to training—so this will be a ‘looser’ lower-bound on human red-team performance (which is the upper-bound on eval awareness).
Maybe—but this does sound like an empirical question about the LLMs capability and its environment and I’d prefer to measure it empirically. You could conservatively assume LLMs know the time, and then if you fail to rule out sufficient awareness, try to take a tighter bound.