It seems pretty challenging to run a something like METR tasks in a non-CoT way. If you are doing tasks like software engineering that are generative (writing any code or agent loop, not just some form of labeling), it’s hard to stop the models from doing things that are essentially CoT. A trick some models[1] will do when writing code is write very verbose in-line comments where they are essentially packing CoT inside the code. Even if you somehow convince them to not write comments, can still do things like write a lot dead code. (It might be possible to control for this somehow though.)
The OP’s methodology sensibly fully side step this by excluding generative software tasks from the main results. One could possibly convert some of METR tasks into labeling-shaped tasks, but not like a straight rerun. Agree could make a version that would be very hard to saturate.
My current impression is the metrics in this study though seem partially saturated already / slightly broken. This is because in the >0.5hr regime there is only a single task on a spikey metric (SHADE-Arena Monitor TPR @ 1% FPR) which is already measuring greater than 50% TPR on frontier models (with massive error bars). Also a lot of the important tasks in the current trend lack actual human data to ground them. (This discuss this some in another other comment and MKodama’s comment). I might be missing something though...
A follow study / iteration which thinks more about good data covering the full spectrum of time seems like would be necessary to draw many conclusions that are trying to be drawn here. Plus future thinking on how timeable things we care about are.
I can’t exactly point here, but loosely especially in the like approx Gemini 2 and GPT-4.1 era models that were RL-ed hard on a bunch of competitive programing problems without much code readability constraints.
It seems pretty challenging to run a something like METR tasks in a non-CoT way. If you are doing tasks like software engineering that are generative (writing any code or agent loop, not just some form of labeling), it’s hard to stop the models from doing things that are essentially CoT. A trick some models[1] will do when writing code is write very verbose in-line comments where they are essentially packing CoT inside the code. Even if you somehow convince them to not write comments, can still do things like write a lot dead code. (It might be possible to control for this somehow though.)
The OP’s methodology sensibly fully side step this by excluding generative software tasks from the main results. One could possibly convert some of METR tasks into labeling-shaped tasks, but not like a straight rerun. Agree could make a version that would be very hard to saturate.
My current impression is the metrics in this study though seem partially saturated already / slightly broken. This is because in the >0.5hr regime there is only a single task on a spikey metric (SHADE-Arena Monitor TPR @ 1% FPR) which is already measuring greater than 50% TPR on frontier models (with massive error bars). Also a lot of the important tasks in the current trend lack actual human data to ground them. (This discuss this some in another other comment and MKodama’s comment). I might be missing something though...
A follow study / iteration which thinks more about good data covering the full spectrum of time seems like would be necessary to draw many conclusions that are trying to be drawn here. Plus future thinking on how timeable things we care about are.
I can’t exactly point here, but loosely especially in the like approx Gemini 2 and GPT-4.1 era models that were RL-ed hard on a bunch of competitive programing problems without much code readability constraints.