I’m a statistician who does some work with METR, and I recently worked on a very similar project to create a Bayesian version of the Time Horizon model. Mine ended up being somewhat different to yours (mine deviates a bit more from the currently structure of the METR model), but its great to see other people stress testing modelling.
On the 80% Time Horizon results I agree that your ‘marginal’ approach is correct, and it is the one I also took in my model. However my 80% results ended up being a factor of 2 Edit:higher than the results of METR’s current model for recent SOTA LLMs. Here is a quick plot I made just after the Opus 4.5 results came out, using the TH1.0 data:
I think there is some natural increase due to how my model’s data is selected however, as my 50% time horizons are also often somewhat higher, and they are mostly within uncertainty bounds anyway:
I’ve taken a very quick look through your code so try and think about the difference, and my guess would be that you find that LLM-difficulty diverges more from log(baseliner_time) time than I do, because you include the tasks with estimated baseliner times when calculating the amount of noise here, whereas I handled tasks with/without baseliner times separately, and only used the former when doing the time horizon calculations.
My definition was: “For a LLM m, I define its ‘p time horizon’ as the delta such that LLM m has (expected) probability p of success on a single attempt at a task with baseliner time delta.” Where we might expect different results for tasks with estimated instead of baselined task lengths because there is effectively another layer of noise added by the estimates.
(I’ll note that out of all the critiques of the Time Horizon work I’m surprised I don’t see more discussion of the tasks which only have estimates, as this seems like one of the most straightforward limitations, and which will only get more relevant as tasks get longer and harder to baseline. Something like only 5⁄30 of the longest tasks currently have baseliner times!)
I’ve love to chat more about Bayesian modelling and general thinking about these kinds of models sometime, and thanks again for the interesting analysis.
Very nice! I’m not able to comment very much since I don’t know the specifics of your model, but can you clarify what you mean by
because you include the tasks with estimated baseliner times when calculating the amount of noise here, whereas I handled tasks with/without baseliner times separately, and only used the former when doing the time horizon calculations.
I have to admit I have worked with the METR data mostly as-is, and not gone into detail about how the times have been estimated. I suppose the problem is that only a subset of the tasks have grounded estimates of human times (as I interpreted HCAST?) and the rest are inferred in a more or less ad-hoc way? If so, then that would explain 80% marginal times being shorter because the residuals would (plausibly) be smaller.
Yes sorry for just dropping in with “I have a model that gives different results” without actually giving the details. I’m trying to get a minimal version of it written up (I had designed it to integrate into METR’s codebase so need to extract it as something that can exits standalone).
Within the runs.json there is a (not especially clearly named) ‘human_source’ field for each row. If this is set to “baseline” then the task length is based on (one or more) human baseliners, if it is “estimate” then it was just estimated without any human actually finishing the task. These estimates are generally quite noisy—I believe somebody told me something like that for some of the tasks where they had both the estimates and the baseliner times, only 60% of the estimates were within a factor of 3 of the (average) baseliner times.
Because you have a unified sigma parameter for how difficulty-for-LLM differs from log(task_length) this ends up incorporating the estimate noise as an additional source of uncertainty. But if you define the p-time-horizon as I did in my first comment as being defined on baselined tasks only then these lead to different results for the 80% time horizons.
Thanks for the great writeup!
I’m a statistician who does some work with METR, and I recently worked on a very similar project to create a Bayesian version of the Time Horizon model. Mine ended up being somewhat different to yours (mine deviates a bit more from the currently structure of the METR model), but its great to see other people stress testing modelling.
On the 80% Time Horizon results I agree that your ‘marginal’ approach is correct, and it is the one I also took in my model. However my 80% results ended up being a factor of 2 Edit:higher than the results of METR’s current model for recent SOTA LLMs. Here is a quick plot I made just after the Opus 4.5 results came out, using the TH1.0 data:
I think there is some natural increase due to how my model’s data is selected however, as my 50% time horizons are also often somewhat higher, and they are mostly within uncertainty bounds anyway:
I’ve taken a very quick look through your code so try and think about the difference, and my guess would be that you find that LLM-difficulty diverges more from log(baseliner_time) time than I do, because you include the tasks with estimated baseliner times when calculating the amount of noise here, whereas I handled tasks with/without baseliner times separately, and only used the former when doing the time horizon calculations.
My definition was: “For a LLM m, I define its ‘p time horizon’ as the delta such that LLM m has (expected) probability p of success on a single attempt at a task with baseliner time delta.” Where we might expect different results for tasks with estimated instead of baselined task lengths because there is effectively another layer of noise added by the estimates.
(I’ll note that out of all the critiques of the Time Horizon work I’m surprised I don’t see more discussion of the tasks which only have estimates, as this seems like one of the most straightforward limitations, and which will only get more relevant as tasks get longer and harder to baseline. Something like only 5⁄30 of the longest tasks currently have baseliner times!)
I’ve love to chat more about Bayesian modelling and general thinking about these kinds of models sometime, and thanks again for the interesting analysis.
Aren’t these a factor of 2 higher than the original METR model?
Good catch! Edited my comment. It had been a while since I had looked at the results and I must have also lost the ability to read in the meantime.
Very nice! I’m not able to comment very much since I don’t know the specifics of your model, but can you clarify what you mean by
I have to admit I have worked with the METR data mostly as-is, and not gone into detail about how the times have been estimated. I suppose the problem is that only a subset of the tasks have grounded estimates of human times (as I interpreted HCAST?) and the rest are inferred in a more or less ad-hoc way? If so, then that would explain 80% marginal times being shorter because the residuals would (plausibly) be smaller.
Yes sorry for just dropping in with “I have a model that gives different results” without actually giving the details. I’m trying to get a minimal version of it written up (I had designed it to integrate into METR’s codebase so need to extract it as something that can exits standalone).
Within the runs.json there is a (not especially clearly named) ‘human_source’ field for each row. If this is set to “baseline” then the task length is based on (one or more) human baseliners, if it is “estimate” then it was just estimated without any human actually finishing the task. These estimates are generally quite noisy—I believe somebody told me something like that for some of the tasks where they had both the estimates and the baseliner times, only 60% of the estimates were within a factor of 3 of the (average) baseliner times.
Because you have a unified sigma parameter for how difficulty-for-LLM differs from log(task_length) this ends up incorporating the estimate noise as an additional source of uncertainty. But if you define the p-time-horizon as I did in my first comment as being defined on baselined tasks only then these lead to different results for the 80% time horizons.