Hyperbolic model fits METR capabilities estimate worse than exponential model

gjm19 Aug 2025 15:12 UTC
201 points
9 comments4 min readLW link

This is a response to https://​​www.lesswrong.com/​​posts/​​mXa66dPR8hmHgndP5/​​hyperbolic-trend-with-upcoming-singularity-fits-metr which claims that a hyperbolic model, complete with an actual singularity in the near future, is a better fit for the METR time-horizon data than a simple exponential model.

I think that post has a serious error in it and its conclusions are the reverse of correct. Hence this one.

(An important remark: although I think Valentin2026 made an important mistake that invalidates his conclusions, I think he did an excellent thing in (1) considering an alternative model, (2) testing it, (3) showing all his working, and (4) writing it up clearly enough that others could check his work. Please do not take any part of this post as saying that Valentin2026 is bad or stupid or any nonsense like that. Anyone can make a mistake; I have made plenty of equally bad ones myself.)

[EDITED to add:] Valentin2026 has now (5) agreed that the thing I think was an error was an error, and (6) edited his post to say so. Thanks, Valentin!

The models

Valentin2026′s post compares the results of fitting two different models to METR’s time-horizon benchmark data (METR’s original blogpost is at https://​​metr.org/​​blog/​​2025-03-19-measuring-ai-ability-to-complete-long-tasks/​​ but the data have been updated since then, most recently as of current writing to account for the GPT-5 launch).

One model, or more accurately one family of models, is hyperbolic: if we let H be the time horizon estimate produced by METR, the model is where is the time at which the estimate shoots off to infinity (“the singularity”, though note that divergence here does not necessarily imply e.g. superintelligence in any very strong sense), A is an overall scale factor, and q is larger when the divergence happens more abruptly. (If you start with a Moore’s-law sort of model of technological progress and then suppose that actually progress gets faster in proportion to H, , then you get a model of this sort with q=1. If progress gets faster in proportion to then you get q=2 and similarly for other values of 2; if it gets faster like then you get something like .)

The other model is a simple Moore’s law sort of model: . Lots of technological things behave kinda like this (but, note, lots of them actually turn out to be sigmoidal: they grow exponentially for a while, then slow down and eventually plateau) and it’s a nice simple default model.

[EDITED to add:] As dr_s points out in a comment, in the absence of exciting new physics nothing can actually grow exponentially for ever. “Lots” should really be “all”, or at least “almost all”, the exceptions being where the end of exponential growth happens in another way like a technology being abandoned in favour of something new and better before its growth slows much. This is probably also a good point at which to say explicitly that a “singularity” in this context should not actually be understood as “the thing becomes infinite” but as “the thing becomes very large in such a way that our model breaks down”.

Valentin2026′s fits

Valentin2026 did some fits of both sorts of models, showed graphs in which the hyperbolic fits looked much better and had smaller mean squared error, and declared that we should maybe shorten our timelines and maybe expect something singularity-like in the near future.

Valentin did something extremely meritorious: he provided both the data and the code that he used to do the fitting. And … looking at the code, it appears that he did something that looks very fishy to me. Here’s the code that fits the hyperbolic model:

q = 2

def model(t, A, t_c):
    return A / (np.power(t_c - t, q))

# Fit model to noisy data
p0 = [1.0, 2027]  # initial guess [A, t_c, q]
bounds = ([0, max(t_50)+0.1], [np.inf, np.inf])
# (make sure t_c > max(t))

popt, pcov = curve_fit(model, t_50, y_50, p0=p0, bounds=bounds, maxfev=20000)

(Documentation for scipy.optimize.curve_fit begins: “Use non-linear least squares to fit a function, f, to data.”)

And here’s the code that fits the exponential model:

logy = np.log(y_50)

# Linear regression: log(y) = log(A) + b * t
coeffs = np.polyfit(t_red, logy, 1)
b = coeffs[0]
logA = coeffs[1]
A = np.exp(logA)

This does a linear least-squares fit to the logarithms of the data points we’re trying to fit.

The problem

This is a perfectly legitimate thing to do, in the abstract. It is an excellent thing to do, if what you care about is the proportional errors in fitting the data. But it is not the same thing as is being done for the hyperbolic model.

When you fit the logs rather than the original data, you will get a fit that when plotted along with the original data looks particularly good when the values are small, and particularly bad when the values are large. A bit like this:

(That’s the exponential fit from Valentin2026′s post.) That certainly looks worse than this hyperbolic fit with q=2:

(As you’d expect, the log-fitted exponential fits better on the left where the numbers are smaller, and worse on the right where they’re bigger.)

And Valentin2026 reports that the total squared error for the exponential fit is much bigger than for the hyperbolic fit—exactly as you would expect when what you’ve fitted is the logarithms and therefore you have larger errors for the larger values. (When I run the code, I get a “total error” of 0.2088 for the hyperbolic fit with q=2, versus 1.2915 for the exponential fit.)

In other words, the apparently worse fit of the exponential model may simply be an artefact of the fact that the hyperbolic and exponential models were fit in fundamentally different ways.

Fixing the problem

If you fit an exponential without doing the logarithm thing, using the same curve_fit function as for the hyperbolic models …

def model(t, b, t0):
   return np.exp(b*(t-t0))
# Fit model to noisy data
p0 = [1.0, 2025]  # initial guess [b, t0]
bounds = ([0, 2000], [np.inf, np.inf])
# (make sure t_c > max(t))
popt, pcov = curve_fit(model, t_50, y_50, p0=p0, bounds=bounds, maxfev=20000)

… then you get this graph

and a “total error” of 0.1067, about 2x better than for the q=2 hyperbolic fit.

I wondered what happens to the hyperbolic model if we fit it in log space. Answer: it’s not great. We get this graph (slightly modified x-axis because otherwise all you get to see is how the model explodes as it approaches its singularity):

and a “total error” of about 6.9, much worse than for either version of the exponential fit.

Conclusion

The hyperbolic model seems (by eye, and by total squared error) to give a worse fit to the METR data than the exponential model, whether we measure errors in linear space or in log space. This doesn’t necessarily mean that the hyperbolic model is wrong, or that the exponential model is right, or that there is nothing singularity-like in the near future, but I think it does mean that whatever updates might have been suggested by Valentin2026′s findings are probably in the wrong direction.

The exponential model does, in fact, seem like a pretty good fit to the data. If you’re going to do short-term extrapolations of AI model time horizons, I think fitting an exponential curve is a decent way to do it.

I would advise extreme caution in extrapolating beyond the short term. A double-exponential fit produces a curve almost indistinguishable by eye from the exponential fit, and a very similar total-error figure, but it predicts much faster growth in the future. A sigmoid fit, , produces an equally good-by-eye fit to the data and (unsurprisingly, given that this model has more parameters) a lower total-error figure, but predicts much slower growth in the future. Extrapolation is a dangerous game unless you have a really good grasp of the underlying mechanisms, which in this case we very much don’t.

gjm19 Aug 2025 15:12 UTC
201 points
9 comments4 min readLW link