I think the central disagreement is about whether AGI/ASI is real or not, and almost everything else is just downstream of that. The claims and policies made in AI 2040 only make sense in a world where you can have AIs that are much much smarter than humans, and that mere AGIs can cause explosive growth.
Given that Seb disagrees with the economics substantially, I would guess that he thinks that AIs will never reach the point where they can fully automate the robot and semiconductor supply chain; if we do get full automation then fast exponential growth seems to quickly fall out of reasonable modeling. Unfortunately I don’t understand his views that well, so it’s hard for me to say exactly where he thinks AI capabilities will cap out.
Re: Transparency and government powers, I think his post pretty badly misrepresented what we were saying, where I think someone who read his piece but not ours would be extremely misled about what sorts of regulations we were proposing. But I also imagine there’s substantial policy disagreement there… so from my perspective it seems like both?
I will say that with government powers in particular, Plan A does involve governments into AI more than the status quo. Insofar as you think that’ll be predictably bad, that is a genuine downside of Plan A. There’s a sense in which any regulation is “central planning”… and yes, we do advocate for certain types of AI regulation. But overall in Plan A we aim for as market based and decentralized solutions: for example, we propose several cap and trade regimes, which seem to me the maximally libertarian way to go about this regulation, and so calling it “central planning” seems pretty misleading. Other regulation is more difficult to set up this way, such as limitations on algorithmic progress.
Thanks for this reply. I do agree that many people who are skeptical of various AI safety proposals don’t really believe in AGI or ASI, but I know that’s not everyone. For example, I am skeptical of AI 2040’s proposals but I very much believe in AGI and ASI, and that’s at the heart of some of the reasons for my concerns. I don’t know if that’s true in Séb’s case: is there a particular thing he wrote which makes you think he does not, or are you speculating/generalizing from experience with others?
Regarding the misrepresentations vs. disagreements, what I would really appreciate is understanding which are which in this post. I count 16 quotes from Séb listed under “False Representations”, but some of them seem more like disagreements to me. It would be helpful if you could clarify which you believe are actual misrepresentations and which you believe are disagreements.
I think it’s true that Séb doesn’t believe in AGI/ASI in the sense that I mean: see his fourth response here. Also see e.g. this model he posted which argues that comparative advantage implies postASI humans will still have jobs (which doesn’t make sense if you think ASI can use the inputs that humans require much more efficiently than the humans do).
Seems very reasonable to be skeptical of many of AI 2040s proposals despite believing in AGI/ASI. I’d be curious to hear more (though perhaps you should post those on the main post, not here, if you feel interested :) ).
Re clarifing the quotes; reading them now I still think they are all misleading but to varying degrees and probably won’t spend more time further getting into it because there’s a lot on my plate right now, but if others think it’d be useful for me or someone else at AIFP to do another pass here trying to further clarify let us know (e.g. by reacting or commenting here).
I don’t think belief in ASI necessarily implies belief in all of:
> (a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response.
Or did you mean some other part of his fourth response?
I also don’t think it requires you believe that no comparative advantage for humans post-ASI exists: positional goods are still a thing, a price premium for human-produced goods is very plausible (and positing that it won’t happen is not a question of capabilities).
I think it’s a common move to claim that people with a different conception of a post-ASI trajectory “don’t believe in” ASI, but when digging into it usually they don’t disagree on raw capabilities, just on what those raw capabilities imply the ASI would be able to do in the world.
(a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response.
As I said in the post, this part is largely a strawman of our views. The part I mostly meant was “Nor do I think we will get real GDP growth of 50% in 2032” and “the model moves far too quickly from AIs being able to perform tasks to robots being reliable, legally deployable, organisationally integrated substitutes for almost all labour, and from there to a closed-loop reproduction of capital”. Perhaps what he means is that there will be delays, but eventually we’ll get 50% GDP growth?
I think it’s a common move to claim that people with a different conception of a post-ASI trajectory “don’t believe in” ASI, but when digging into it usually they don’t disagree on raw capabilities, just on what those raw capabilities imply the ASI would be able to do in the world.
I tend to find the exact opposite is true. People love claiming that the real disagreement is real world bottlenecks, but usually its really differences in capability expectations. I think the AIs will be wildly superintelligent, with vast quantities operating at the equivalent of 100x or 1000x human speeds, also with a much higher qualitative intelligence due to things like knowing way more than any human can know, having much more experience than any human can every get, and having a physically much larger and more connected brain such that they are able to make discoveries and model things that are far too complicated for humans.
This is obviously an extreme milestone; we outline earlier milestones here: https://ai-rates-calculator.vercel.app/, for example. I would encourage people to make forecasts of when they think these specific milestones will be crossed (which might be “in hundreds of years or never”). If earlier than that, maybe its just a timelines / takeoff speed disagreement.
I kind of expect that many people would argue that this is the wrong ontology for thinking about AI capability progressions, and that actually, it’s somehow going to be more diffuse/multipolar or something, in a way where thinking in these terms isn’t useful for modeling the world, in which case I’d love to hear a better frame.
Still, I think that most people with very different views would tend to disagree that the notion of ” superintelligent” AI that I outlined above would happen but will have small effects on the world. I think this view is extremely hard to make coherent. Once we’ve got AIs like that, the cognitive labour supply would become enourmous, such that almost all the cognitive labour happening would be AIs, not humans.
Totally appreciate you guys have a lot on your plate right now! At least two of the examples I mentioned in my top-level comment didn’t seem like misrepresentations to me (though two of them did), which makes me confused as to how much of this post is accurate if it really is saying that all of the quoted examples are misrepresentations (hence the reason for my comment requesting that clarification). I haven’t carefully dug into every claim myself but overall my sense is that this post very heavily blurs the line between vehement disagreement and actual misrepresentation such that I don’t feel there’s a clear takeaway for me about how much of each is going on in Séb’s critique.
Thanks for this comment!
I think the central disagreement is about whether AGI/ASI is real or not, and almost everything else is just downstream of that. The claims and policies made in AI 2040 only make sense in a world where you can have AIs that are much much smarter than humans, and that mere AGIs can cause explosive growth.
Given that Seb disagrees with the economics substantially, I would guess that he thinks that AIs will never reach the point where they can fully automate the robot and semiconductor supply chain; if we do get full automation then fast exponential growth seems to quickly fall out of reasonable modeling. Unfortunately I don’t understand his views that well, so it’s hard for me to say exactly where he thinks AI capabilities will cap out.
Re: Transparency and government powers, I think his post pretty badly misrepresented what we were saying, where I think someone who read his piece but not ours would be extremely misled about what sorts of regulations we were proposing. But I also imagine there’s substantial policy disagreement there… so from my perspective it seems like both?
I will say that with government powers in particular, Plan A does involve governments into AI more than the status quo. Insofar as you think that’ll be predictably bad, that is a genuine downside of Plan A. There’s a sense in which any regulation is “central planning”… and yes, we do advocate for certain types of AI regulation. But overall in Plan A we aim for as market based and decentralized solutions: for example, we propose several cap and trade regimes, which seem to me the maximally libertarian way to go about this regulation, and so calling it “central planning” seems pretty misleading. Other regulation is more difficult to set up this way, such as limitations on algorithmic progress.
Thanks for this reply. I do agree that many people who are skeptical of various AI safety proposals don’t really believe in AGI or ASI, but I know that’s not everyone. For example, I am skeptical of AI 2040’s proposals but I very much believe in AGI and ASI, and that’s at the heart of some of the reasons for my concerns. I don’t know if that’s true in Séb’s case: is there a particular thing he wrote which makes you think he does not, or are you speculating/generalizing from experience with others?
Regarding the misrepresentations vs. disagreements, what I would really appreciate is understanding which are which in this post. I count 16 quotes from Séb listed under “False Representations”, but some of them seem more like disagreements to me. It would be helpful if you could clarify which you believe are actual misrepresentations and which you believe are disagreements.
I think it’s true that Séb doesn’t believe in AGI/ASI in the sense that I mean: see his fourth response here. Also see e.g. this model he posted which argues that comparative advantage implies postASI humans will still have jobs (which doesn’t make sense if you think ASI can use the inputs that humans require much more efficiently than the humans do).
Seems very reasonable to be skeptical of many of AI 2040s proposals despite believing in AGI/ASI. I’d be curious to hear more (though perhaps you should post those on the main post, not here, if you feel interested :) ).
Re clarifing the quotes; reading them now I still think they are all misleading but to varying degrees and probably won’t spend more time further getting into it because there’s a lot on my plate right now, but if others think it’d be useful for me or someone else at AIFP to do another pass here trying to further clarify let us know (e.g. by reacting or commenting here).
I don’t think belief in ASI necessarily implies belief in all of:
> (a) extremely fast diffusion and societal transformation, (b) a view that all profits accrue maximally to the labs, (c) that the prescriptions advanced in the essay (like expropriations and forced IP diffusion) have minimal impacts on said profits; (d) the claim that you get explosive GDP growth very soon, (e) that ‘de facto’ nationalisation and profit redistribution through UBI is an optimal response.
Or did you mean some other part of his fourth response?
I also don’t think it requires you believe that no comparative advantage for humans post-ASI exists: positional goods are still a thing, a price premium for human-produced goods is very plausible (and positing that it won’t happen is not a question of capabilities).
I think it’s a common move to claim that people with a different conception of a post-ASI trajectory “don’t believe in” ASI, but when digging into it usually they don’t disagree on raw capabilities, just on what those raw capabilities imply the ASI would be able to do in the world.
As I said in the post, this part is largely a strawman of our views. The part I mostly meant was “Nor do I think we will get real GDP growth of 50% in 2032” and “the model moves far too quickly from AIs being able to perform tasks to robots being reliable, legally deployable, organisationally integrated substitutes for almost all labour, and from there to a closed-loop reproduction of capital”. Perhaps what he means is that there will be delays, but eventually we’ll get 50% GDP growth?
I tend to find the exact opposite is true. People love claiming that the real disagreement is real world bottlenecks, but usually its really differences in capability expectations. I think the AIs will be wildly superintelligent, with vast quantities operating at the equivalent of 100x or 1000x human speeds, also with a much higher qualitative intelligence due to things like knowing way more than any human can know, having much more experience than any human can every get, and having a physically much larger and more connected brain such that they are able to make discoveries and model things that are far too complicated for humans.
This is obviously an extreme milestone; we outline earlier milestones here: https://ai-rates-calculator.vercel.app/, for example. I would encourage people to make forecasts of when they think these specific milestones will be crossed (which might be “in hundreds of years or never”). If earlier than that, maybe its just a timelines / takeoff speed disagreement.
I kind of expect that many people would argue that this is the wrong ontology for thinking about AI capability progressions, and that actually, it’s somehow going to be more diffuse/multipolar or something, in a way where thinking in these terms isn’t useful for modeling the world, in which case I’d love to hear a better frame.
Still, I think that most people with very different views would tend to disagree that the notion of ” superintelligent” AI that I outlined above would happen but will have small effects on the world. I think this view is extremely hard to make coherent. Once we’ve got AIs like that, the cognitive labour supply would become enourmous, such that almost all the cognitive labour happening would be AIs, not humans.
Totally appreciate you guys have a lot on your plate right now! At least two of the examples I mentioned in my top-level comment didn’t seem like misrepresentations to me (though two of them did), which makes me confused as to how much of this post is accurate if it really is saying that all of the quoted examples are misrepresentations (hence the reason for my comment requesting that clarification). I haven’t carefully dug into every claim myself but overall my sense is that this post very heavily blurs the line between vehement disagreement and actual misrepresentation such that I don’t feel there’s a clear takeaway for me about how much of each is going on in Séb’s critique.