Yes, unless I messed up, METR’s code runs a logistic regression of log2(task duration) against success probability, so my model predicts a raw fitted coefficient (the second column in the table) close to -ln(2) ≈ −0.69.
Oh right sorry I missed the derivation that among nM+nT samples, the maximum is equally likely to be any of them and so the probability that the largest number from the model the largest of them is
nMnM+nT=11+nT/nM=11+exp(−(lognM−lognT))
This model then predicts that models “ELO ratings” - lognM would grow linearly over time, which (based on this chart GPT5 gave me) I think corresponds roughly with the progress in chess from 2007 onwards
It also makes the quantitative prediction that a doubling in compute (or compute efficiency) leads to a 2⁄3 win probability, or around 120 Elo points. (Credit to the Hex paper for this observation.) Under 18-month doublings (per one version of Moore’s law), this would be around 800 Elo points per decade, which looks like a bit of an overestimate but similar to the fastest observed rate of progress.
In figure 5 the X axis is log time horizon and not time horizon—does this fit with your model?
Yes, unless I messed up, METR’s code runs a logistic regression of log2(task duration) against success probability, so my model predicts a raw fitted coefficient (the second column in the table) close to -ln(2) ≈ −0.69.
Oh right sorry I missed the derivation that among nM+nT samples, the maximum is equally likely to be any of them and so the probability that the largest number from the model the largest of them is
nMnM+nT=11+nT/nM=11+exp(−(lognM−lognT))
This model then predicts that models “ELO ratings” - lognM would grow linearly over time, which (based on this chart GPT5 gave me) I think corresponds roughly with the progress in chess from 2007 onwards
It also makes the quantitative prediction that a doubling in compute (or compute efficiency) leads to a 2⁄3 win probability, or around 120 Elo points. (Credit to the Hex paper for this observation.) Under 18-month doublings (per one version of Moore’s law), this would be around 800 Elo points per decade, which looks like a bit of an overestimate but similar to the fastest observed rate of progress.