However, I’m quite skeptical of this type of consideration making a big difference because the ML industry has already varied the compute input massively, with over 7 OOMs of compute difference between research now (in 2025) vs at the time of AlexNet 12 years ago, (invalidating the view that there is some relatively narrow range of inputs in which neither input is bottlenecking)
Seems like this is a strawman of the bottlenecks view, which would say that the number of near frontier experiments, not compute, is the bottleneck and this quantity didn’t scale up over that time
ETA: for example, if the compute scale up had happened, but no one had been allowed to run experiments with more compute than AlexNet, it seems a lot more plausible that the compute would have stopped helping because there just wouldn’t have been enough people to plan the experiments
Plus the claim that alg progress might have been actively enabled by the access to new hardware scales
Toby Ord’s recent work digs into this. For the case of OAI, i think it suggests most capability improvements are coming from continuing the curve for longer, but there is also some effect from a steeper curve
https://x.com/tobyordoxford/status/1999870642032967987?s=20