Regardless, most definitions [of compute overhang] are not very analytically useful or decision-relevant. As of April 2023, the cost of compute for an LLM’s final training run is around $40M. This is tiny relative to the value of big technology companies, around $1T. I expect compute for training models to increase dramatically in the next few years; this would cause how much more compute labs could use if they chose to to decrease.
I think this is just another way of saying there is a very large compute overhang now and it is likely to get at least somewhat smaller over the next few years.
Keep in mind that “hardware overhang” first came about when we had no idea if we would figure out how to make AGI before or after we had the compute to implement it.
I have the impression that for reasons I don’t fully understand, scaling up training compute isn’t just a matter of being willing to spend more. One does not simply spend $1B on compute.
Ideas and training compute substitute for each other sufficiently well enough that I don’t think it’s useful to talk about “[figuring] out how to make AGI before or after we [have] the compute to implement it.” (And when “‘hardware overhang’ first came about” it had very different usage, e.g. the AI Impacts definition.)
I think this is just another way of saying there is a very large compute overhang now and it is likely to get at least somewhat smaller over the next few years.
Keep in mind that “hardware overhang” first came about when we had no idea if we would figure out how to make AGI before or after we had the compute to implement it.
Agree in part.
I have the impression that for reasons I don’t fully understand, scaling up training compute isn’t just a matter of being willing to spend more. One does not simply spend $1B on compute.
Ideas and training compute substitute for each other sufficiently well enough that I don’t think it’s useful to talk about “[figuring] out how to make AGI before or after we [have] the compute to implement it.” (And when “‘hardware overhang’ first came about” it had very different usage, e.g. the AI Impacts definition.)