Interesting to see. Some things I wonder about, (1) which tasks most influence the results, (2) how much time numbers make sense, and (3) general elicitation questions
Which tasks and kinds of tasks are most influencing the headline? With METR’s timeline plots there has been discussion here on LW and elsewhere about if certain items have a large influence on the results. I was wondering which kinds of tasks most explain the recent doublings already, and what might worlds look like where they doubled again according to this methodology.
I sort of attempted to vibe very rough versions of this analysis. It’s hard without the underlying data, but can get claude to try to extract accuracy numbers from Tables 7-13 and pixel peep Fig 47 to get medians and IQRs for the human-time data.[1]
Things in the ~1-3min band are most likely to be near the 50% crossover point and I think will have more influence on the regression. A few outliers like the “vibe-coding sabotage” on the Y axis and the SHADE monitor on the X axis might deserve some consideration. These are both use a kind of TPR @ ~1-2% FPR metric, different than some other datasets. Unclear effects here.
This is vibey though and would want access to underlying data to do more confident analysis.
There’s also other ways of quantify the influence of each dataset (where the “vibe-coding sabaotage” task again shows up as a clear outlier with more than twice as much influence as any other dataset). I hide it in a collapsible though as I’m not super confident in methodology.
Vibey can we calculate an influence score?
Claude’s explanation:
The bars estimate how much each benchmark plausibly contributed to the headline time-horizon gain between GPT-4o and GPT-5.5. The headline metric is the 50% crossover of a logistic curve fit to success vs. log human solve time, and a basic property of logistic fits is that an observation’s influence on the fitted curve scales with p(1−p) — maximal for tasks the model passes about half the time, near zero for tasks it almost always passes or almost always fails. Since the crossover moved from roughly 0.9 to 3 minutes between these two models, accuracy gains on tasks in that time range supply the most evidence about where the crossover sits. Each bar is therefore the benchmark’s accuracy change (chance-corrected, approximately following the paper’s normalization — exactly for MCQ tasks, while the detection tasks’ ~1.5% baseline was treated as 0, a ~1pp difference) multiplied by a p(1−p) weight centered at ~1.6 minutes and about a decade wide in log time, averaged over the benchmark’s per-question human-time distribution as extracted from the paper’s Figure 47. The whiskers show how that weight varies across the middle 50% of each benchmark’s tasks. This is a heuristic approximation, not a refit of the paper’s actual estimator: it captures the main channel (gains near the crossover move the crossover) but ignores how off-crossover gains rotate the curve’s slope, and ignores the anchoring effect of long tasks still near 0%, which constrain how far the crossover can move. The ranking at the top is stable across reasonable choices of kernel width, but the precise values shouldn’t be taken seriously — a leave-one-benchmark-out refit on the underlying data would be the right version of this analysis.
Time numbers
In general I have mixed thoughts on human time estimates as a metric in some cases, and how well time-horizon can approximate what I’d be most concerned about models packing into a forward pass[2]
It seems worth reflecting on how much trust to put on time-horizon metrics if we often can’t actually get good human-time numbers in the first place. One particular consideration is many of the most influential datasets in the regression don’t have actual human time measurements, and instead come from this process of using Claude to estimate human time. I think I’d believe claude can make good guesses here (and have numbers here for the tested tasks), but still seems slightly worth considering . The authors are thorough in this paper, and I eventually noticed Fig 19 in appendix which gives the regression only looking at cases with true numbers. The double time goes from 373 [167,691] days → 349 [166, 1883] days. So the point estimate is surprisingly stable, but a confidence interval from ~6 months to 5 years is substantial.
Again, I like the ideas here and the thoroughness of the paper, but just bringing up general consideration given how much attention and extrapolation METR’s original time horizon plot got. I might eventually form a clearer view here and write up more thoughts…
Edit: see also the discussion on MKodama’s comment about the logistic plots. The fact there is only single task in the >0.5hr regime looks pretty problematic for much of the main analysis (eg, for Fig 2 here / fig 5 paper). If going to make some of the claims above there really should be either more tasks in this time regime, or this part of the plots should be excluded or better indicate it’s just a single one task. I haven’t thought as much about the o3-mini anchor, but a different possibly reasonable angle.
What’s going on with few shot?
Language models are always sensitive to how you prompt them. This is especially true for earlier ones. In the headline regression GPT-2 and GPT-3 are excluded. This might be reasonable, but also worth considering aspects like prompt construction. GPT-3 was quite a smart model, it just was not “ChatGPT 100M user product-ready” because it required more care when prompting. Section 3 discusses some about how few-shot is used to elicit the non-CoT format. It’s unclear how many few-shot examples are used or if this is typically enough for GPT-3-type base models. This question doesn’t seem core to the study, but maybe a separate question would study a version of this data with uniform like 10-shot in every dataset or something.
--
Thanks for all the work on exploring these models and sharing findings.
btw, I think the caption on fig 47 might be off, since it says IQR, but appears to show all points? below whiskers assumes taking only an approximate middle slice of fig 47 points
stuff like Choi et al’s measurements on the models making inferences of user attributes. These don’t have clean mappings to how long I think a human would take to perform, but still fit concerning categories of “intuition, not reasoning” aspects that happen in a forward pass
The mainline trends and uncertainty estimates are computed from bootstraps which resample many times over different benchmarks, meaning that the analysis is robust to leaving out any particular (subset of) benchmarks. This is true for both the main doubling trends and the individual model TH estimates. From the paper:
>The fact there is only single task in the >0.5hr regime looks pretty problematic
We have lots of tasks that are >0.5 hr. In the main trend we only include a subset of (mostly shorter) tasks, but including all tasks doesn’t change the trend that much (the overall doubling times reduces a bit because frontier models can do some long horizon SWE tasks without CoT, e.g., just outputting tool calls at each step).
From the paper:
> Figure 22: Comparison of TH trend lines using all tasks including short-answer, generation, and multi-turn agentic. The most significant impact of including the multi-turn agentic tasks is an increase in the point-estimate THs for the latest frontier models which perform very well on these tasks.
It would be helpful if you could ask more precise questions about the rest :)
Right, bootstrapping here seems like a good approach, and appreciate the thoroughness here. It will communicate an interval, but there is still this wonder about what does the current doubling look like and what might future ones would look like. There’s various taxonomies here (eg, which tasks can gain on knowledge retrieval over reasoning, as you mentioned in the post). I tried this some in the fig I made.
Bootstrapping is a tool that tells us some things, but not others. It will still be grounded in the selected distribution of tasks (eg, in a cartoon case where bootstrapping can mislead us, imagine if we selected 10⁄32 tasks that were at the core about memorizing say birthdays of baseball players. If models suddenly got good at memorizing birthdays, it would look like a large improvement with a tight CI improvement, but wouldn’t capture overall variance of core metric. Again, cartoon extreme). So this is something was curious about after the initial read of the paper.
In some concrete terms here, an individual dataset will have a ~64% chance[1]of being included during bootstrapping. I was just trying to get an impression of this. Can understand that more times than not outlier-ish datasets like “Vibe-code Sabatoge” and “SHADE Arena-Monitor” will be included. I shared some findings in my comment, but as I said vibey and still underexplored for future work.
> Over 0.5hr regime
Right, to be clear I think focusing on a short-answer subset is the cleanest way to get at this non-CoT question. But when this filtering is where it gets somewhat problematic when trying to understand fitting a logistic regression for a given model in the 0 to 4hr range (like in post Fig 2), and when thinking about what will future doublings under the methodology have to look like as push upwards.
As you highlight when including non-short answer tasks for non-headline results, yes there are ~7 rather than 1, so more coverage (sorry if felt like I was ignoring this in a unfair way). I feel like I don’t have much certainty on how CoT-free the other 6 tasks are (which my impression is y’all are also a bit uncertain of, given filter out). As I mentioned in the comment elsewhere with Raemon, I know models, especially of a certain era, were very prone to doing things like writing long in-line comments, and basically packing CoT into the middle of their answers.
I agree that the fact that the regressions with and without these tasks doesn’t change much is reassuring. I still think the questions around dataset influence seem important. I felt like one thing I learned when looking at this was an appreciation for how the tails of logistic regression matter less (counter to my intuitions on linear regression where tail points matter). So then it becomes about looking at what are the most influential datasets near the crossover point as I tried to loosely explore above, where datasets that are near the crossover point will most influence the results. I didn’t make a version of my plot using the non-short-answer sample yet to try to see measures of influence there. Thanks for the out-of-band exchange sharing the data if did later.
> It would be helpful if you could ask more precise questions about the rest :)
I don’t have a quick expansion to give here right now as direct questions (regarding thoughts on time horizon as a metric, how estimateable it is on things we care about, elicitation variations). Possibly might expand later, or happy to chat synchronously sometime.
--
Thanks for engaging with this! I like many aspects of the study, and hopefully discussion is useful.
Thank you for your comments! Just following on from the point about individual dataset influence, we ran a leave-one-out study, sharing those results below. You are correct that SHADE and Vibe-coding (amongst a few others) have a strong impact, but as you can see these don’t materially change the conclusions we landed on around doubling time trends and associated uncertainty. We will add this to the paper, thank you pointing it out.
>The fact there is only single task in the >0.5hr regime looks pretty problematic
I also wanted to add that, as part of the bootstrap, we are incorporating uncertainty in solve times meaning that even if the point-estimate solve times for some questions is <0.5hr, it could still contribute in the bootstrap at >0.5hr. (this is just to say, there are more tasks in the bucket than it might seem!).
Hi Dewi, thanks for the reply. This is interesting.
Sorry I realized I messed up in my top level comment where I incorrectly vibed the subselection of datasets used (I don’t think it was specified in the paper which 32 datasets are included or not, and Claude assumed in a way that looked reasonable enough but was off on a few). Sorry about that. I’m glad to your new figure exists.
Note though, ideally we’d want to ablate SHADE CoT and SHADE Action together (and maybe the two a-level datasets?) as these are very related tasks (I think?). Keeping them separate for like SHADE monitor makes it look like there are two unique tasks in this high end regime.
I attempted to make a new version tweaked version of my plot using the jsons you sent.
Things with large LOO Δ above generally have either have a tall change in this lollipop plot or are important parts of the tails (eg, arithmetic is important for flushing out the low end. Removing drops that whole area closer to the “tower of london” number).
Things in this center area would have to greatly change in worlds of the next measured doubling.
This is not including the possible extra expansion on the ICR due to the variance in the estimates them self. I’m not currently sure how to capture this. Some things like a-level MCQ has no IQR I think because all labeled at 1min.
> but as you can see these don’t materially change the conclusions we landed on around doubling time trends and associated uncertainty
Right, this does seem pretty robust to various slicing of this particular dataset of tasks. I wanted to start the thread with intuition building of what that particular set is and what’s changing. I feel personally now have a bit better sense of this.
The “doesn’t materially change” part should make sure we are truely internalize the associated uncertainty ranging from months to years. Similarly in the MKodama’s comment when looking at the logistic regression plots on the capabilities of a given model make sure we appreciate the error bars.
It still feels like we currently have a pretty confused signal for a lot of this span, especially in the upper half. The logistic fit will be rough. This is also already communicated by the individual estimates on the frontier models spanning 2 OOM from ~20 seconds to ~1 hour, so seems good as long as people internalize that to the extent.
All things to think about as the community thinks about for future work in the area, as well as making sure the community doesn’t over interpret the lowest nuance “373 day doubling time currently at 3min” version. The paper/post conclusion points around models capable of twenty-five minutes of no-CoT reasoning by 2030 are going need different methodology to measure (there’s not task diversity in this regime). Plus thinking on how well time horizon is a way to think about things we care about here.
Interesting to see. Some things I wonder about, (1) which tasks most influence the results, (2) how much time numbers make sense, and (3) general elicitation questions
Which tasks and kinds of tasks are most influencing the headline?
With METR’s timeline plots there has been discussion here on LW and elsewhere about if certain items have a large influence on the results. I was wondering which kinds of tasks most explain the recent doublings already, and what might worlds look like where they doubled again according to this methodology.
I sort of attempted to vibe very rough versions of this analysis. It’s hard without the underlying data, but can get claude to try to extract accuracy numbers from Tables 7-13 and pixel peep Fig 47 to get medians and IQRs for the human-time data.[1]
Things in the ~1-3min band are most likely to be near the 50% crossover point and I think will have more influence on the regression. A few outliers like the “vibe-coding sabotage” on the Y axis and the SHADE monitor on the X axis might deserve some consideration. These are both use a kind of TPR @ ~1-2% FPR metric, different than some other datasets. Unclear effects here.
This is vibey though and would want access to underlying data to do more confident analysis.
There’s also other ways of quantify the influence of each dataset (where the “vibe-coding sabaotage” task again shows up as a clear outlier with more than twice as much influence as any other dataset). I hide it in a collapsible though as I’m not super confident in methodology.
Vibey can we calculate an influence score?
Time numbers
In general I have mixed thoughts on human time estimates as a metric in some cases, and how well time-horizon can approximate what I’d be most concerned about models packing into a forward pass[2]
It seems worth reflecting on how much trust to put on time-horizon metrics if we often can’t actually get good human-time numbers in the first place. One particular consideration is many of the most influential datasets in the regression don’t have actual human time measurements, and instead come from this process of using Claude to estimate human time. I think I’d believe claude can make good guesses here (and have numbers here for the tested tasks), but still seems slightly worth considering . The authors are thorough in this paper, and I eventually noticed Fig 19 in appendix which gives the regression only looking at cases with true numbers. The double time goes from 373 [167,691] days → 349 [166, 1883] days. So the point estimate is surprisingly stable, but a confidence interval from ~6 months to 5 years is substantial.
Again, I like the ideas here and the thoroughness of the paper, but just bringing up general consideration given how much attention and extrapolation METR’s original time horizon plot got. I might eventually form a clearer view here and write up more thoughts…
Edit: see also the discussion on MKodama’s comment about the logistic plots. The fact there is only single task in the >0.5hr regime looks pretty problematic for much of the main analysis (eg, for Fig 2 here / fig 5 paper). If going to make some of the claims above there really should be either more tasks in this time regime, or this part of the plots should be excluded or better indicate it’s just a single one task. I haven’t thought as much about the o3-mini anchor, but a different possibly reasonable angle.
What’s going on with few shot?
Language models are always sensitive to how you prompt them. This is especially true for earlier ones. In the headline regression GPT-2 and GPT-3 are excluded. This might be reasonable, but also worth considering aspects like prompt construction. GPT-3 was quite a smart model, it just was not “ChatGPT 100M user product-ready” because it required more care when prompting. Section 3 discusses some about how few-shot is used to elicit the non-CoT format. It’s unclear how many few-shot examples are used or if this is typically enough for GPT-3-type base models. This question doesn’t seem core to the study, but maybe a separate question would study a version of this data with uniform like 10-shot in every dataset or something.
--
Thanks for all the work on exploring these models and sharing findings.
btw, I think the caption on fig 47 might be off, since it says IQR, but appears to show all points? below whiskers assumes taking only an approximate middle slice of fig 47 points
stuff like Choi et al’s measurements on the models making inferences of user attributes. These don’t have clean mappings to how long I think a human would take to perform, but still fit concerning categories of “intuition, not reasoning” aspects that happen in a forward pass
Quick response:
> (1) which tasks most influence the results
The mainline trends and uncertainty estimates are computed from bootstraps which resample many times over different benchmarks, meaning that the analysis is robust to leaving out any particular (subset of) benchmarks. This is true for both the main doubling trends and the individual model TH estimates. From the paper:
>The fact there is only single task in the >0.5hr regime looks pretty problematic
We have lots of tasks that are >0.5 hr. In the main trend we only include a subset of (mostly shorter) tasks, but including all tasks doesn’t change the trend that much (the overall doubling times reduces a bit because frontier models can do some long horizon SWE tasks without CoT, e.g., just outputting tool calls at each step).
From the paper:
> Figure 22: Comparison of TH trend lines using all tasks including short-answer, generation, and multi-turn agentic. The most significant impact of including the multi-turn agentic tasks is an increase in the point-estimate THs for the latest frontier models which perform very well on these tasks.
It would be helpful if you could ask more precise questions about the rest :)
Thanks for reply!
> which tasks most influence the results
Right, bootstrapping here seems like a good approach, and appreciate the thoroughness here. It will communicate an interval, but there is still this wonder about what does the current doubling look like and what might future ones would look like. There’s various taxonomies here (eg, which tasks can gain on knowledge retrieval over reasoning, as you mentioned in the post). I tried this some in the fig I made.
Bootstrapping is a tool that tells us some things, but not others. It will still be grounded in the selected distribution of tasks (eg, in a cartoon case where bootstrapping can mislead us, imagine if we selected 10⁄32 tasks that were at the core about memorizing say birthdays of baseball players. If models suddenly got good at memorizing birthdays, it would look like a large improvement with a tight CI improvement, but wouldn’t capture overall variance of core metric. Again, cartoon extreme). So this is something was curious about after the initial read of the paper.
In some concrete terms here, an individual dataset will have a ~64% chance[1]of being included during bootstrapping. I was just trying to get an impression of this. Can understand that more times than not outlier-ish datasets like “Vibe-code Sabatoge” and “SHADE Arena-Monitor” will be included. I shared some findings in my comment, but as I said vibey and still underexplored for future work.
> Over 0.5hr regime
Right, to be clear I think focusing on a short-answer subset is the cleanest way to get at this non-CoT question. But when this filtering is where it gets somewhat problematic when trying to understand fitting a logistic regression for a given model in the 0 to 4hr range (like in post Fig 2), and when thinking about what will future doublings under the methodology have to look like as push upwards.
As you highlight when including non-short answer tasks for non-headline results, yes there are ~7 rather than 1, so more coverage (sorry if felt like I was ignoring this in a unfair way). I feel like I don’t have much certainty on how CoT-free the other 6 tasks are (which my impression is y’all are also a bit uncertain of, given filter out). As I mentioned in the comment elsewhere with Raemon, I know models, especially of a certain era, were very prone to doing things like writing long in-line comments, and basically packing CoT into the middle of their answers.
I agree that the fact that the regressions with and without these tasks doesn’t change much is reassuring. I still think the questions around dataset influence seem important. I felt like one thing I learned when looking at this was an appreciation for how the tails of logistic regression matter less (counter to my intuitions on linear regression where tail points matter). So then it becomes about looking at what are the most influential datasets near the crossover point as I tried to loosely explore above, where datasets that are near the crossover point will most influence the results. I didn’t make a version of my plot using the non-short-answer sample yet to try to see measures of influence there. Thanks for the out-of-band exchange sharing the data if did later.
> It would be helpful if you could ask more precise questions about the rest :)
I don’t have a quick expansion to give here right now as direct questions (regarding thoughts on time horizon as a metric, how estimateable it is on things we care about, elicitation variations). Possibly might expand later, or happy to chat synchronously sometime.
--
Thanks for engaging with this! I like many aspects of the study, and hopefully discussion is useful.
Sampling 32 tasks with replacement.
Thank you for your comments! Just following on from the point about individual dataset influence, we ran a leave-one-out study, sharing those results below. You are correct that SHADE and Vibe-coding (amongst a few others) have a strong impact, but as you can see these don’t materially change the conclusions we landed on around doubling time trends and associated uncertainty. We will add this to the paper, thank you pointing it out.
>The fact there is only single task in the >0.5hr regime looks pretty problematic
I also wanted to add that, as part of the bootstrap, we are incorporating uncertainty in solve times meaning that even if the point-estimate solve times for some questions is <0.5hr, it could still contribute in the bootstrap at >0.5hr. (this is just to say, there are more tasks in the bucket than it might seem!).
Hi Dewi, thanks for the reply. This is interesting.
Sorry I realized I messed up in my top level comment where I incorrectly vibed the subselection of datasets used (I don’t think it was specified in the paper which 32 datasets are included or not, and Claude assumed in a way that looked reasonable enough but was off on a few). Sorry about that. I’m glad to your new figure exists.
Note though, ideally we’d want to ablate SHADE CoT and SHADE Action together (and maybe the two a-level datasets?) as these are very related tasks (I think?). Keeping them separate for like SHADE monitor makes it look like there are two unique tasks in this high end regime.
I attempted to make a new version tweaked version of my plot using the jsons you sent.
Things with large LOO Δ above generally have either have a tall change in this lollipop plot or are important parts of the tails (eg, arithmetic is important for flushing out the low end. Removing drops that whole area closer to the “tower of london” number).
Things in this center area would have to greatly change in worlds of the next measured doubling.
This is not including the possible extra expansion on the ICR due to the variance in the estimates them self. I’m not currently sure how to capture this. Some things like a-level MCQ has no IQR I think because all labeled at 1min.
> but as you can see these don’t materially change the conclusions we landed on around doubling time trends and associated uncertainty
Right, this does seem pretty robust to various slicing of this particular dataset of tasks. I wanted to start the thread with intuition building of what that particular set is and what’s changing. I feel personally now have a bit better sense of this.
The “doesn’t materially change” part should make sure we are truely internalize the associated uncertainty ranging from months to years. Similarly in the MKodama’s comment when looking at the logistic regression plots on the capabilities of a given model make sure we appreciate the error bars.
It still feels like we currently have a pretty confused signal for a lot of this span, especially in the upper half. The logistic fit will be rough. This is also already communicated by the individual estimates on the frontier models spanning 2 OOM from ~20 seconds to ~1 hour, so seems good as long as people internalize that to the extent.
All things to think about as the community thinks about for future work in the area, as well as making sure the community doesn’t over interpret the lowest nuance “373 day doubling time currently at 3min” version. The paper/post conclusion points around models capable of twenty-five minutes of no-CoT reasoning by 2030 are going need different methodology to measure (there’s not task diversity in this regime). Plus thinking on how well time horizon is a way to think about things we care about here.