Returned to this post today. When I first encountered this, I thought “this seems like just a more vague version of the METR graph”. But actually the ECI has a lot of advantages.
The method is great because it only requires partial eval coverage. If I’m comparing a collection of models, I can just select some set of evals which cover them all and then merge them. I can’t compare time horizons for models in a meaningful way.
The METR graph has started to fail now because the time horizons are too long. ECI is still going
There’s no vagueness over 50% vs 80% vs 99% time horizons
ECI is better than time horizons if you want to measure general capabilities just because there’s so much more, and more diverse, data. If AI progress is linear, it’s much more likely for ECI to be linear than for log time horizon to be linear. The only disasvantage is that the Y axis is not automatically interpretable unless you’re familiar with ECI.
Returned to this post today. When I first encountered this, I thought “this seems like just a more vague version of the METR graph”. But actually the ECI has a lot of advantages.
The method is great because it only requires partial eval coverage. If I’m comparing a collection of models, I can just select some set of evals which cover them all and then merge them. I can’t compare time horizons for models in a meaningful way.
The METR graph has started to fail now because the time horizons are too long. ECI is still going
There’s no vagueness over 50% vs 80% vs 99% time horizons
Nice work, Epoch.
ECI is better than time horizons if you want to measure general capabilities just because there’s so much more, and more diverse, data. If AI progress is linear, it’s much more likely for ECI to be linear than for log time horizon to be linear. The only disasvantage is that the Y axis is not automatically interpretable unless you’re familiar with ECI.