Just by looking at the benchmark scores you can see that it is very off-trend. But of course error bars for such long time horizons (even done with much better methodology than mine) are huge.
You should also put ever-decreasing credence in reported time horizons, cf Ryan’s post
The old METR time horizon benchmark has mostly saturated when it comes to measuring 50%-reliability time-horizon (as in, scores are sufficiently high this measurement is unreliable), but at 80% reliability the best publicly deployed models are at a bit over an hour while I expect the best internal models are reaching a bit below 2 hours. I expect that increasingly this 80%-reliability score is dominated by relatively niche tasks that don’t centrally reflect automating software engineering or AI R&D. Further, the time horizon measurement is increasingly sensitive to the task distribution.
Hm, this is comparing a model release date with a model preview internal release date. I suspect that Opus 4.6 had a internal preview release data that was earlier than Feb 5th. So you probably need to add some kind of fudge factor for that to get an accurate doubling period.
Opus 4.6 released on Feb 5. Mythos Preview sort of released today, if you count the system card. Your results impute a ~5.7x time horizon jump.
That’s 61 days, which works out to a (61/log2(5.7)) = about 24 days doubling period.
METR’s most recent doubling period was 4.3 months, AI Futures’ most recent one is 4. This would be about .8 months.
That would be very off-trend if true, to say the least.
We should expect a step change in model size and pretraining compute to be off trend.
Wouldn’t that imply that Opus 4.6 should have been below trend since it wasn’t a step change in model size or pretraining (as far as I know).
Yes, that is implied.
Just by looking at the benchmark scores you can see that it is very off-trend. But of course error bars for such long time horizons (even done with much better methodology than mine) are huge.
You should also put ever-decreasing credence in reported time horizons, cf Ryan’s post
Hm, this is comparing a model release date with a model preview internal release date. I suspect that Opus 4.6 had a internal preview release data that was earlier than Feb 5th. So you probably need to add some kind of fudge factor for that to get an accurate doubling period.