As StanislavKyrm has written elsewhere, we have indeed tested Claude Code and Codex, and previously, we’ve done the same with other agent scaffolds in the literature.
Onto the other two problems you raise:
Problem 1: The METR eval produces results with egregious confidence intervals, and the METR chart misleadingly hides this.
I don’t know if I’d say “egregious”. I agree the confidence intervals are large, but since they’re largely a result of uncertainties in the task suite, the differences between models are generally significant. Probably the real fix is a better way to present this uncertainty.
Problem 2: There’s a lack of sample size for long duration tasks.
Yep, real problem. Opus 4.6′s “real” 50% time horizon is accurately described as “beyond the ability of METR to accurately measure with the Time Horizon 1.1 Suite”.
The good news is that this problem is easy to fix; just add more longer-duration tasks. METR has the money for this. Seriously, why has this not been done?
We’re working on it!
For example, we made a suite of tasks in collaboration with Epoch (MirrorCode), that unfortunately turned out to be 1) less representative of the tasks currently in the suite and 2) seems to have been partially saturated while it was being developed.
Thanks for writing this!
As StanislavKyrm has written elsewhere, we have indeed tested Claude Code and Codex, and previously, we’ve done the same with other agent scaffolds in the literature.
Onto the other two problems you raise:
I don’t know if I’d say “egregious”. I agree the confidence intervals are large, but since they’re largely a result of uncertainties in the task suite, the differences between models are generally significant. Probably the real fix is a better way to present this uncertainty.
Yep, real problem. Opus 4.6′s “real” 50% time horizon is accurately described as “beyond the ability of METR to accurately measure with the Time Horizon 1.1 Suite”.
We’re working on it!
For example, we made a suite of tasks in collaboration with Epoch (MirrorCode), that unfortunately turned out to be 1) less representative of the tasks currently in the suite and 2) seems to have been partially saturated while it was being developed.
As I’ve written elsewhere, it’s pretty hard to make long tasks, and it’s not just a monetary expense.