Backing up, there is a basic point that I think The Problem is making, that I think is solid and I’m curious if you agree with. Paraphrasing: Many people underestimate the danger of superhuman AI because they mistakenly believe that skilled humans are close to the top of the range of mental ability in most domains.
I agree with this point and think it is an important point. However, I also interpret “The Problem” as arguing “AIs will go from weak to extremely capable very quickly”. This is argued for both in section 1 and under “ASI would be able to destroy us” and in the bullet I highlighted (as well as some of the other bullets around this). In practice, takeoff speeds is actually an important disagreement which has a big effect on the prognosis and the most effective interventions for lowering risk. (I think it’s probably somewhat of a crux for the bottom line recommendation being a good one, especially after taking into account feasibility constraints and is even more of a crux in thinking that this bottom line recommendation is the best recommendation to advocate for.)
There are, I believe many readers who would glaze over a toned-down version of the text that will correctly internalize the severity of the mistake when it’s presented in a bombastic way. Punchy text can also be fun to read, which matters.
I agree that it’s key to make it very clear to readers the ways in which the text is arguing for claims they currently disagree with so that readers don’t round off the claims to something much weaker (which is less surprising to them or more normal seeming).[1]
But, I also think it’s important to avoid false claims especially if they will seem saliently wrong to otherwise persuadable readers! People react poorly to invalid arguments used to argue for a conclusion they currently disagree with (IMO for justified reasons)!
If I’m reading you right, you’re saying something like: AI capabilities have been growing at a pace in most domains where the time between “can do at all” and “vastly outstrips humans” takes at least years and sometimes decades, and it is importantly wrong to characterize this as “very soon afterwards.”
Yep, this is what I was saying. (Note that the original text also emphasizes “As soon as” rather than “very soon”, so I think the actual situation is especially misleading.)
I notice that I’m confused about whether you think this is importantly wrong in the sense of invalidating the basic point that people neglect how much room there is above humans in cognitive domains, or whether you think it’s importantly wrong because it conflicts with other aspects of the basic perspective such as takeoff speeds and the importance of slowing down before we have AGI vs muddling through. Or maybe you’re just arguing that it’s hyperbolic, and you just wish the language was softer?
I don’t think it invalidates the point that “people neglect how much room there is above humans in cognitive domains”, but it seems importantly wrong because takeoff speeds matter a bunch and this makes a false claim arguing for faster takeoff speeds. I think this essay implicitly operates as though takeoff speeds are fast in various places (and directly makes claims related to takeoff speeds like this one!).
Suppose that I simply agree. Should we re-write the paragraph to say something like “AI systems routinely outperform humans in narrow domains. When AIs become at all competitive with human professionals on a given task, humans usually cease to be able to compete within just a handful of years. It would be unexpected if this pattern suddenly stopped applying for all the tasks that AI can’t yet compete with human professionals on.”? Do you agree that the core point would remain, if we did that rewrite? How would you feel about a simple footnote that says “Yes, we’re being hyperbolic here, but have you noticed the skulls of people who thought machines would not outstrip humans?”
I don’t like the footnote (especially because “notice the skulls” isn’t a well known expression). The rewrite seems reasonable to me. (I appreciate moving from “can do X” to “tasks” which seems like a better way to think about this if you want to include cost/speed dominance.) It seems fine to make this more punchy by emphasizing how far AIs sometimes outstrip humans.
This seems especially important because a common thing in AI communication by AI companies and other political actors is saying something in a way where it can be interpreted both very weakly by people who aren’t very bought into AI being massively transformative while also being consistent with a view that AI is more dangerous/important/transformative that’s being emphasized to other actors. For one of the clearest examples of this, see Jared Kaplan’s senate testimony which includes lines like “One key concern is the possibility that an advanced AI may develop harmful emergent behaviors, such as deception or strategic planning abilities.” that can easily be rounded off to something much, much weaker than “AIs which are as capable or much more capable than humans (including AIs we plan on building) might plot against humanity by default, possibly leading to a violent AI takeover” (which is much closer to the “key concern” people at Anthropic are actually worried about). Notably, Anthropic now uses “country of geniuses” and a specific description of AI capabilities rather than “advanced AI” because they want to more clearly communicate the level of capability rather than maintaining strategic ambiguity and allowing themselves to be rounded down. (Which seems like a good comms change on my views.)
I agree with this point and think it is an important point. However, I also interpret “The Problem” as arguing “AIs will go from weak to extremely capable very quickly”. This is argued for both in section 1 and under “ASI would be able to destroy us” and in the bullet I highlighted (as well as some of the other bullets around this). In practice, takeoff speeds is actually an important disagreement which has a big effect on the prognosis and the most effective interventions for lowering risk. (I think it’s probably somewhat of a crux for the bottom line recommendation being a good one, especially after taking into account feasibility constraints and is even more of a crux in thinking that this bottom line recommendation is the best recommendation to advocate for.)
I agree that it’s key to make it very clear to readers the ways in which the text is arguing for claims they currently disagree with so that readers don’t round off the claims to something much weaker (which is less surprising to them or more normal seeming).[1]
But, I also think it’s important to avoid false claims especially if they will seem saliently wrong to otherwise persuadable readers! People react poorly to invalid arguments used to argue for a conclusion they currently disagree with (IMO for justified reasons)!
Yep, this is what I was saying. (Note that the original text also emphasizes “As soon as” rather than “very soon”, so I think the actual situation is especially misleading.)
I don’t think it invalidates the point that “people neglect how much room there is above humans in cognitive domains”, but it seems importantly wrong because takeoff speeds matter a bunch and this makes a false claim arguing for faster takeoff speeds. I think this essay implicitly operates as though takeoff speeds are fast in various places (and directly makes claims related to takeoff speeds like this one!).
I don’t like the footnote (especially because “notice the skulls” isn’t a well known expression). The rewrite seems reasonable to me. (I appreciate moving from “can do X” to “tasks” which seems like a better way to think about this if you want to include cost/speed dominance.) It seems fine to make this more punchy by emphasizing how far AIs sometimes outstrip humans.
This seems especially important because a common thing in AI communication by AI companies and other political actors is saying something in a way where it can be interpreted both very weakly by people who aren’t very bought into AI being massively transformative while also being consistent with a view that AI is more dangerous/important/transformative that’s being emphasized to other actors. For one of the clearest examples of this, see Jared Kaplan’s senate testimony which includes lines like “One key concern is the possibility that an advanced AI may develop harmful emergent behaviors, such as deception or strategic planning abilities.” that can easily be rounded off to something much, much weaker than “AIs which are as capable or much more capable than humans (including AIs we plan on building) might plot against humanity by default, possibly leading to a violent AI takeover” (which is much closer to the “key concern” people at Anthropic are actually worried about). Notably, Anthropic now uses “country of geniuses” and a specific description of AI capabilities rather than “advanced AI” because they want to more clearly communicate the level of capability rather than maintaining strategic ambiguity and allowing themselves to be rounded down. (Which seems like a good comms change on my views.)