Very nice! I’m not able to comment very much since I don’t know the specifics of your model, but can you clarify what you mean by
because you include the tasks with estimated baseliner times when calculating the amount of noise here, whereas I handled tasks with/without baseliner times separately, and only used the former when doing the time horizon calculations.
I have to admit I have worked with the METR data mostly as-is, and not gone into detail about how the times have been estimated. I suppose the problem is that only a subset of the tasks have grounded estimates of human times (as I interpreted HCAST?) and the rest are inferred in a more or less ad-hoc way? If so, then that would explain 80% marginal times being shorter because the residuals would (plausibly) be smaller.
Yes sorry for just dropping in with “I have a model that gives different results” without actually giving the details. I’m trying to get a minimal version of it written up (I had designed it to integrate into METR’s codebase so need to extract it as something that can exits standalone).
Within the runs.json there is a (not especially clearly named) ‘human_source’ field for each row. If this is set to “baseline” then the task length is based on (one or more) human baseliners, if it is “estimate” then it was just estimated without any human actually finishing the task. These estimates are generally quite noisy—I believe somebody told me something like that for some of the tasks where they had both the estimates and the baseliner times, only 60% of the estimates were within a factor of 3 of the (average) baseliner times.
Because you have a unified sigma parameter for how difficulty-for-LLM differs from log(task_length) this ends up incorporating the estimate noise as an additional source of uncertainty. But if you define the p-time-horizon as I did in my first comment as being defined on baselined tasks only then these lead to different results for the 80% time horizons.
Very nice! I’m not able to comment very much since I don’t know the specifics of your model, but can you clarify what you mean by
I have to admit I have worked with the METR data mostly as-is, and not gone into detail about how the times have been estimated. I suppose the problem is that only a subset of the tasks have grounded estimates of human times (as I interpreted HCAST?) and the rest are inferred in a more or less ad-hoc way? If so, then that would explain 80% marginal times being shorter because the residuals would (plausibly) be smaller.
Yes sorry for just dropping in with “I have a model that gives different results” without actually giving the details. I’m trying to get a minimal version of it written up (I had designed it to integrate into METR’s codebase so need to extract it as something that can exits standalone).
Within the runs.json there is a (not especially clearly named) ‘human_source’ field for each row. If this is set to “baseline” then the task length is based on (one or more) human baseliners, if it is “estimate” then it was just estimated without any human actually finishing the task. These estimates are generally quite noisy—I believe somebody told me something like that for some of the tasks where they had both the estimates and the baseliner times, only 60% of the estimates were within a factor of 3 of the (average) baseliner times.
Because you have a unified sigma parameter for how difficulty-for-LLM differs from log(task_length) this ends up incorporating the estimate noise as an additional source of uncertainty. But if you define the p-time-horizon as I did in my first comment as being defined on baselined tasks only then these lead to different results for the 80% time horizons.