Nice analysis. I can’t add anything substantive, but this writeup crystallized for me just how much we’re all focusing on METR’s horizon lengths work. On the one hand, it’s the best data set we have at the moment for quantitative extrapolation, so of course we should focus on it. On the other hand, it’s only one data set, and could easily turn out to not imply what we think it implies.
My only points are (a) we shouldn’t weight the horizon length trends too heavily, and (b) boy do we need additional metrics that are both extrapolatable, and plausibly linked to actual outcomes of interest.
Nice analysis. I can’t add anything substantive, but this writeup crystallized for me just how much we’re all focusing on METR’s horizon lengths work. On the one hand, it’s the best data set we have at the moment for quantitative extrapolation, so of course we should focus on it. On the other hand, it’s only one data set, and could easily turn out to not imply what we think it implies.
My only points are (a) we shouldn’t weight the horizon length trends too heavily, and (b) boy do we need additional metrics that are both extrapolatable, and plausibly linked to actual outcomes of interest.