Any given benchmark is going to be a limited indicator, and will look like it is accelerating as models move into the steep part of the S-curve. I recommend looking at the Epoch Capabilities Index, which is an aggregate of many benchmarks. It shows models through Astra as basically on trend.
Generally, the capabilities are more or less on trend.
But we care about the specific subset of capabilities, namely being helpful in innovative, cutting edge AI research. Because that is what matters for RSI.
Our benchmarks usually don’t measure this too well (in particular, no fixed benchmark can express innovative, cutting edge AI research, because this is a moving target).
In this sense, we would like something public to complement the internal reports from the labs on how they are using AI to accelerate their AI R&D. In this sense, what we are seeing with this particular competition seems to be compatible with recent reports claiming considerable AI R&D acceleration. So I am planning to monitor this one for some weeks.
(This measures not the model capability as such, but the ability of an AI research group to make progress using the best available model.)
Any given benchmark is going to be a limited indicator, and will look like it is accelerating as models move into the steep part of the S-curve. I recommend looking at the Epoch Capabilities Index, which is an aggregate of many benchmarks. It shows models through Astra as basically on trend.
Generally, the capabilities are more or less on trend.
But we care about the specific subset of capabilities, namely being helpful in innovative, cutting edge AI research. Because that is what matters for RSI.
Our benchmarks usually don’t measure this too well (in particular, no fixed benchmark can express innovative, cutting edge AI research, because this is a moving target).
In this sense, we would like something public to complement the internal reports from the labs on how they are using AI to accelerate their AI R&D. In this sense, what we are seeing with this particular competition seems to be compatible with recent reports claiming considerable AI R&D acceleration. So I am planning to monitor this one for some weeks.
(This measures not the model capability as such, but the ability of an AI research group to make progress using the best available model.)