Thanks for making this, it’s good to see high-effort critiques.
Let’s start with a plot that shouldn’t be too surprising. Four reasonable models fit the METR data equally well. They agree about the past but disagree strongly about the future.
Note that we have data going back to 2019 (GPT-2), and it looks like the other models wouldn’t fit the earlier trend well. (The raw data should be in the time horizons 1.0 folder). I’d guess the best fitting models over the whole range would be linear and a slightly superlinear power law.
METR also cares about 80% success and fits a separate model for that.
Actually it’s worse than that; we just get the 80% point of the logistic. Initially 80% success was an afterthought, but since it’s clear people care about it, Alex Barry has been doing Bayesian analysis that supports it.
But there are actually two reasonable ways to define “80% success,” and they give different answers.
Typical: Pick a task of average difficulty for its length. Can the model solve it 80% of the time? This is roughly what METR computes.
Marginal: Pick a random task of that length. What’s the expected success rate? Because some tasks are much harder than average, the hard ones drag down the average more than easy ones push it up.
Our method is actually closer to computing “Marginal”, because when fitting the logistic curve, the points near x = 1 hour include all trials for all tasks estimated at that length. We didn’t actually have the means to get the “Typical” 80% point since we don’t compute “average difficulty for its length” anywhere.
In your analysis, the “Typical” 80% time horizon of the last point (I think Claude 4.5 Opus) is around 110 minutes, whereas ours is 42 minutes. I’m not sure what makes the marginal so low in your analysis (5 minutes) but Alex suggests it might be that effect you mentioned. Ideally, I think the curves would look something like this:
Basically there are two factors that change the original analysis vs “typical”
We have limited baseline data (often 0-2 baselines per task), so our time estimates will be noisy estimates of the average human time.
Tasks that take humans the same average time will still differ in difficulty (this is “typical” vs “marginal”).
If we were able to account for each of the factors or estimate them using stats, our data model would get more predictive, the slopes increase, and the 80% time horizon increase from our results → “marginal” → “typical”. But like you say we actually want to stop at “marginal”.
I chatted with Thomas a bit about this, and I also agree that the default METR model should also output things that are close to the ‘marginal’ definition of time horizon (or at least as well as it can be approximated with the inverse logit sigmoid).
I think the important thing to realise is that while one needs to take additional steps for the ‘marginal’ approach when fitting a model that explicitly accounts for the deviation in task-length-for-humans vs task-difficulty-for-llms, models that don’t explicitly account for this (such as the original METR model) should have it naturally learned into the shape of their logistic curve.
(A similar thing is also true for having the discrimination parameter vary by task instead of by model—if it varies by task this uncertainty needs to be accounted for in the time horizon calculations, but since this is not the case in the original METR model it does not).
I think the important thing to realise is that while one needs to take additional steps for the ‘marginal’ approach when fitting a model that explicitly accounts for the deviation in task-length-for-humans vs task-difficulty-for-llms, models that don’t explicitly account for this (such as the original METR model) should have it naturally learned into the shape of their logistic curve.
I don’t immediately see this. The marginal idea is roughly about integrating over random effects, and that’s hard to capture without actually doing it. My statement that METR’s original approach is about the typical effect is wrong though.
I think we agree and I just stated this badly—I was just meaning to say that METR’s original approach is closer to marginal despite them not explicitly doing the integrating over the random effects (although I agree you need to do integrate over the random effects in models that include them to get the marginal time horizon).
Note that we have data going back to 2019 (GPT-2), and it looks like the other models wouldn’t fit the earlier trend well. (The raw data should be in the time horizons 1.0 folder). I’d guess the best fitting models over the whole range would be linear and a slightly superlinear power law.
Thanks, I didn’t think of checking of this! I refitted the models to all the data in the updated post. The story doesn’t really change much when it comes to fit. I also think it’s ok, probably preferable, to use only v1.1. There’s not much reason to go back this far, especially due to the well-known kink which happens roughly at the start of v1.1 anyway.
Actually it’s worse than that; we just get the 80% point of the logistic.
Hmmm… Yeah, I gotta admit I didn’t really check how you did this. I’m not sure what this means in practice and would careful interpreting it. It doesn’t clearly map to either typical or marginal. Marginal seems intuitively out of the question because it requires integration over random effects though. It’s just something else entirely, no?
Ideally, I think the curves would look something like this: [...]
I had Claude make some plots at the updated post that shows the corresponding curves in my model. I don’t have a strong opinion of what the plots should ideally look like, as its kinda open how even perfectly estimated human times () maps to equivalent llm-times. (Which I take it you agree with looking at your linked post on “Reasons time horizon is overrated and misinterpreted”?)
Anyway, the difference between marginal and typical in my plots are pretty well-explained from the residuals plot, as it has to take into account the scattering of the residuals (and we have a sizable proportion of tasks with equivalent time 22x as large!). Is this the close to the “true marginal”? I mean, yes, it is, IF we take human completion time in the data set as being true. If we don’t, then we have to model it.
To estimate the marginal stuff properly we need to model the baseline times. I didn’t really try this, and I do think a good implementation would use all the available data, not only the columns on the Github. But it is possible to slap a lognormal on the public data too, but one data point per task is not enough to gain a lot of information, and our results will be very prior-driven. And honestly I think that’s where we are wrt to 80% horizons anyway.
Thanks for making this, it’s good to see high-effort critiques.
Note that we have data going back to 2019 (GPT-2), and it looks like the other models wouldn’t fit the earlier trend well. (The raw data should be in the time horizons 1.0 folder). I’d guess the best fitting models over the whole range would be linear and a slightly superlinear power law.
Actually it’s worse than that; we just get the 80% point of the logistic. Initially 80% success was an afterthought, but since it’s clear people care about it, Alex Barry has been doing Bayesian analysis that supports it.
Our method is actually closer to computing “Marginal”, because when fitting the logistic curve, the points near x = 1 hour include all trials for all tasks estimated at that length. We didn’t actually have the means to get the “Typical” 80% point since we don’t compute “average difficulty for its length” anywhere.
In your analysis, the “Typical” 80% time horizon of the last point (I think Claude 4.5 Opus) is around 110 minutes, whereas ours is 42 minutes. I’m not sure what makes the marginal so low in your analysis (5 minutes) but Alex suggests it might be that
Basically there are two factors that change the original analysis vs “typical”
We have limited baseline data (often 0-2 baselines per task), so our time estimates will be noisy estimates of the average human time.
Tasks that take humans the same average time will still differ in difficulty (this is “typical” vs “marginal”).
If we were able to account for each of the factors or estimate them using stats, our data model would get more predictive, the slopes increase, and the 80% time horizon increase from our results → “marginal” → “typical”. But like you say we actually want to stop at “marginal”.
I chatted with Thomas a bit about this, and I also agree that the default METR model should also output things that are close to the ‘marginal’ definition of time horizon (or at least as well as it can be approximated with the inverse logit sigmoid).
I think the important thing to realise is that while one needs to take additional steps for the ‘marginal’ approach when fitting a model that explicitly accounts for the deviation in task-length-for-humans vs task-difficulty-for-llms, models that don’t explicitly account for this (such as the original METR model) should have it naturally learned into the shape of their logistic curve.
(A similar thing is also true for having the discrimination parameter vary by task instead of by model—if it varies by task this uncertainty needs to be accounted for in the time horizon calculations, but since this is not the case in the original METR model it does not).
I don’t immediately see this. The marginal idea is roughly about integrating over random effects, and that’s hard to capture without actually doing it. My statement that METR’s original approach is about the typical effect is wrong though.
I think we agree and I just stated this badly—I was just meaning to say that METR’s original approach is closer to marginal despite them not explicitly doing the integrating over the random effects (although I agree you need to do integrate over the random effects in models that include them to get the marginal time horizon).
Thanks, I didn’t think of checking of this! I refitted the models to all the data in the updated post. The story doesn’t really change much when it comes to fit. I also think it’s ok, probably preferable, to use only v1.1. There’s not much reason to go back this far, especially due to the well-known kink which happens roughly at the start of v1.1 anyway.
Hmmm… Yeah, I gotta admit I didn’t really check how you did this. I’m not sure what this means in practice and would careful interpreting it. It doesn’t clearly map to either typical or marginal. Marginal seems intuitively out of the question because it requires integration over random effects though. It’s just something else entirely, no?
I had Claude make some plots at the updated post that shows the corresponding curves in my model. I don’t have a strong opinion of what the plots should ideally look like, as its kinda open how even perfectly estimated human times (
Anyway, the difference between marginal and typical in my plots are pretty well-explained from the residuals plot, as it has to take into account the scattering of the residuals (and we have a sizable proportion of tasks with equivalent time 22x as large!). Is this the close to the “true marginal”? I mean, yes, it is, IF we take human completion time in the data set as being true. If we don’t, then we have to model it.
To estimate the marginal stuff properly we need to model the baseline times. I didn’t really try this, and I do think a good implementation would use all the available data, not only the columns on the Github. But it is possible to slap a lognormal on the public data too, but one data point per task is not enough to gain a lot of information, and our results will be very prior-driven. And honestly I think that’s where we are wrt to 80% horizons anyway.