But if the brain is already near said practical physical limits, then merely achieving brain parity in AGI at all will already require using up most of the optimizational slack, leaving not much left for a hard takeoff—thus a slower takeoff.
While you do talk about stuff related to this in the post / I’m not sure you disagree about facts, I think I want to argue about interpretation / frame.
That is, efficiency is a numerator over a denominator; I grant that we’re looking at the right numerator, but even if human brains are maximally efficient by denominator 1, they might be highly inefficient by denominator 2, and the core value of AI may be being able to switch from denominator 1 to denominator 2 (rather than being a ‘straightforward upgrade’).
The analogy between birds and planes is probably useful here; birds are (as you would expect!) very efficient at miles flown per calorie, but if it’s way easier to get ‘calories’ through chemical engineering on petroleum, then a less efficient plane that consumes jet fuel can end up cheaper. And if what’s economically relevant is “top speed” or “time it takes to go from New York to London”, then planes can solidly beat birds. I think we were living in the ‘fast takeoff’ world for planes (in a technical instead of economic sense), even tho this sort of reasoning would have suggested there would be slow takeoff as we struggled to reach bird efficiency.
The easiest disanalogy between humans and computers is probably “ease of adding more watts”; my brain is running at ~10W because it was ‘designed’ in an era when calories were super-scarce and cooling was difficult. But electricity is super cheap, and putting 200W through my GPU and then dumping it into my room costs basically nothing. (Once you have ‘datacenter’ levels of compute, electricity and cooling costs are significant; but again substantially cheaper than the costs of feeding similar numbers of humans.)
A second important disanalogy is something like “ease of adding more compute in parallel”; if I want to add a second GPU to my computer, this is a mild hassle and only takes some tweaks to work; if I want to add a second brain to my body, this is basically impossible. [This is maybe underselling humans, who make organizations to ‘add brains’ in this way, but I think this is still probably quite important for timeline-related concerns.]
I discuss some of that in [this comment] in reply to Steven Byrnes. I agree electricity is cheap, and discuss that. But electricity is not free, and still becomes a constraint.
In the end of the article I discuss/estimate near future brain-scale AGI requiring 1000 GPUs for 1000 brain size agents in parallel, using roughly 1MW total or 1KW per agent instance. That works out to about $2,000/yr per agent for the power&cooling cost. Or if we just estimate directly based on vast.ai prices it’s more like $5,000/yr per agent total for hardware rental (including power costs). The rental price using enterprise GPUs is at least 4x as much, so more like $20,000/yr per agent. So the potential economic advantage is not yet multiple OOM. It’s actually more like little to no advantage for low-end robotic labor, or perhaps 1 OOM advantage for programmers/researchers/ec. But if we had AGI today GPU prices would just skyrocket to arbitrage that advantage, at least until foundries could ramp up GPU production.
So anyway given some bound/estimate for power cost per agent, this does allow us to roughly bound the total amount of AGI compute near term achievable, as both world power production and foundry output is difficult to ramp up rapidly.
$2,000/yr per agent is nothing, when we are talking about hypothetical AGI. This seems to be evidence against your claim that energy is a taut constraint.
Sure, the actual price of compute would be more, because of the hardware and facilities etc. But that doesn’t change the bottom line that energy is not a taut constraint.
Maybe you are saying that in the future energy will become a taut constraint because we can’t make chips significantly more energy efficient but we can make them significantly cheaper in every other way, so energy will become the dominant part of the cost of compute?
Energy is always an engineering constraint: it’s a primary constraint on Moore’s Law, and thus also a primary limiter on a fast takeoff with GPUs (because world power supply isn’t enough to support net ANN compute much larger than current brain population net compute).
But again I already indicated it’s probably not a ‘taut constraint’ on early AGI in terms of economic cost—at least in my model of likely requirements for early not-smarter-than-human AGI.
Also yes additionally longer term we can expect energy to become a larger fraction of economic cost—through some combination of more efficient chip production, or just the slowing of moore’s law itself (which implies chips holding value for much longer, thus reducing the dominant hardware depreciation component of rental costs)
While you do talk about stuff related to this in the post / I’m not sure you disagree about facts, I think I want to argue about interpretation / frame.
That is, efficiency is a numerator over a denominator; I grant that we’re looking at the right numerator, but even if human brains are maximally efficient by denominator 1, they might be highly inefficient by denominator 2, and the core value of AI may be being able to switch from denominator 1 to denominator 2 (rather than being a ‘straightforward upgrade’).
The analogy between birds and planes is probably useful here; birds are (as you would expect!) very efficient at miles flown per calorie, but if it’s way easier to get ‘calories’ through chemical engineering on petroleum, then a less efficient plane that consumes jet fuel can end up cheaper. And if what’s economically relevant is “top speed” or “time it takes to go from New York to London”, then planes can solidly beat birds. I think we were living in the ‘fast takeoff’ world for planes (in a technical instead of economic sense), even tho this sort of reasoning would have suggested there would be slow takeoff as we struggled to reach bird efficiency.
The easiest disanalogy between humans and computers is probably “ease of adding more watts”; my brain is running at ~10W because it was ‘designed’ in an era when calories were super-scarce and cooling was difficult. But electricity is super cheap, and putting 200W through my GPU and then dumping it into my room costs basically nothing. (Once you have ‘datacenter’ levels of compute, electricity and cooling costs are significant; but again substantially cheaper than the costs of feeding similar numbers of humans.)
A second important disanalogy is something like “ease of adding more compute in parallel”; if I want to add a second GPU to my computer, this is a mild hassle and only takes some tweaks to work; if I want to add a second brain to my body, this is basically impossible. [This is maybe underselling humans, who make organizations to ‘add brains’ in this way, but I think this is still probably quite important for timeline-related concerns.]
I discuss some of that in [this comment] in reply to Steven Byrnes. I agree electricity is cheap, and discuss that. But electricity is not free, and still becomes a constraint.
In the end of the article I discuss/estimate near future brain-scale AGI requiring 1000 GPUs for 1000 brain size agents in parallel, using roughly 1MW total or 1KW per agent instance. That works out to about $2,000/yr per agent for the power&cooling cost. Or if we just estimate directly based on vast.ai prices it’s more like $5,000/yr per agent total for hardware rental (including power costs). The rental price using enterprise GPUs is at least 4x as much, so more like $20,000/yr per agent. So the potential economic advantage is not yet multiple OOM. It’s actually more like little to no advantage for low-end robotic labor, or perhaps 1 OOM advantage for programmers/researchers/ec. But if we had AGI today GPU prices would just skyrocket to arbitrage that advantage, at least until foundries could ramp up GPU production.
So anyway given some bound/estimate for power cost per agent, this does allow us to roughly bound the total amount of AGI compute near term achievable, as both world power production and foundry output is difficult to ramp up rapidly.
$2,000/yr per agent is nothing, when we are talking about hypothetical AGI. This seems to be evidence against your claim that energy is a taut constraint.
Sure, the actual price of compute would be more, because of the hardware and facilities etc. But that doesn’t change the bottom line that energy is not a taut constraint.
Maybe you are saying that in the future energy will become a taut constraint because we can’t make chips significantly more energy efficient but we can make them significantly cheaper in every other way, so energy will become the dominant part of the cost of compute?
Energy is always an engineering constraint: it’s a primary constraint on Moore’s Law, and thus also a primary limiter on a fast takeoff with GPUs (because world power supply isn’t enough to support net ANN compute much larger than current brain population net compute).
But again I already indicated it’s probably not a ‘taut constraint’ on early AGI in terms of economic cost—at least in my model of likely requirements for early not-smarter-than-human AGI.
Also yes additionally longer term we can expect energy to become a larger fraction of economic cost—through some combination of more efficient chip production, or just the slowing of moore’s law itself (which implies chips holding value for much longer, thus reducing the dominant hardware depreciation component of rental costs)
Or maybe you aren’t saying energy is a taut constraint at all? It sure sounded like you did but maybe I misinterpreted you.