I agree that anchoring stuff to release dates isn’t perfect because the underlying variable of “how long does it take until a model is released” is variable, but I think is variability is sufficiently low that it doesn’t cause that much of an issue in practice. The trend is only going to be very solid over multiple model releases and it won’t reliably time things to within 6 months, but that seems fine to me.
I agree that if you add one outlier data point and then trend extrapolate between just the last two data points, you’ll be in trouble, but fortunately, you can just not do this and instead use more than 2 data points.
This also means that I think people shouldn’t update that much on the individual o3 data point in either direction. Let’s see where things go for the next few model releases.
That one seems to work more reliably, perhaps because it became the metric the industry aims for.
I agree that if you add one outlier data point and then trend extrapolate between just the last two data points, you’ll be in trouble
My issue here is that there wasn’t that much variance in the performance of all preceding models they benchmarked: from GPT-2 to Sonnet 3.7, they seem to almost perfectly fall on the straight line. Then, the very first advancement of the frontier after the trend-model is released is an outlier. That suggests an overfit model.
I do agree that it might just be a coincidental outlier and that we should wait and see whether the pattern recovers with subsequent model releases. But this is suspicious enough I feel compelled to make my prediction now.
Do you also dislike Moore’s law?
I agree that anchoring stuff to release dates isn’t perfect because the underlying variable of “how long does it take until a model is released” is variable, but I think is variability is sufficiently low that it doesn’t cause that much of an issue in practice. The trend is only going to be very solid over multiple model releases and it won’t reliably time things to within 6 months, but that seems fine to me.
I agree that if you add one outlier data point and then trend extrapolate between just the last two data points, you’ll be in trouble, but fortunately, you can just not do this and instead use more than 2 data points.
This also means that I think people shouldn’t update that much on the individual o3 data point in either direction. Let’s see where things go for the next few model releases.
That one seems to work more reliably, perhaps because it became the metric the industry aims for.
My issue here is that there wasn’t that much variance in the performance of all preceding models they benchmarked: from GPT-2 to Sonnet 3.7, they seem to almost perfectly fall on the straight line. Then, the very first advancement of the frontier after the trend-model is released is an outlier. That suggests an overfit model.
I do agree that it might just be a coincidental outlier and that we should wait and see whether the pattern recovers with subsequent model releases. But this is suspicious enough I feel compelled to make my prediction now.