If tasks are independent from each other, then it might be possible much sooner, and perhaps even now (with a separate subagent to evaluate each task).
It’s probably more the case that they don’t trust the models enough yet to do a sensitive task of this kind correctly on their own. A human wants at least to double-check and certify the numbers (given how influential these particular numbers are).
If running a METR evaluation on a frontier model takes 25-60 hours of staff time, the models themselves will be able to do it for them by 2027.
If tasks are independent from each other, then it might be possible much sooner, and perhaps even now (with a separate subagent to evaluate each task).
It’s probably more the case that they don’t trust the models enough yet to do a sensitive task of this kind correctly on their own. A human wants at least to double-check and certify the numbers (given how influential these particular numbers are).
Yes, an alternate framing would be “METR has one held-out task with a time horizon of 25-60 hours”.