mostly it does not match my practical experience so far
I mostly wouldn’t expect it to at this point, FWIW. The people engaged right now are by and large people sincerely grappling with the idea, and particularly people who are already bought into takeover risk. Whereas one of the main mechanisms by which I expect misuse of the idea is that people who are uncomfortable with the concept of “AI takeover” can still classify themselves as part of the AI safety coalition when it suits them.
As an illustration of this happening to Paul’s worldview, see this Vox article titled “AI disaster won’t look like the Terminator. It’ll be creepier.” My sense is that both Paul and Vox wanted to distance themselves from Eliezer’s scenarios, and so Paul phrased his scenario in a way which downplayed stuff like “robot armies” and then Vox misinterpreted Paul to further downplay that stuff. (More on this from Carl here.) Another example: Sam Altman has previously justified racing to AGI by appealing to the idea that a slow takeoff is better than a fast takeoff.
Now, some of these dynamics are unavoidable—we shouldn’t stop debating takeoffs just because people might misuse the concepts. But it’s worth keeping an eye out for ideas that are particularly prone to this, and gradual disempowerment seems like one.
in practical terms, gradual disempowerment does not seem particularly convenient set of ideas for justifying that working in an AGI company on something very prosaic which helps the company is the best thing to do.
Well, it’s much more convenient than “AI takeover”, and so the question is how much people are motivated to use it to displace the AI takeover meme in their internal narratives.
when trying to support thinking about the problems, we use understanding-directed labels/pointers (Post-AGI Civilizational Equilibria), even though in many ways it could have been easier to use GD as a brand.
Kudos for doing so. I don’t mean to imply that you guys are unaware of this issue or negligent; IMO it’s a pretty hard problem to avoid. I agree that stuff like “understanding power” is nowhere near adequate as a replacement. However, I do think that there’s some concept like “empowering humans” which is a way to address both takeover risk and gradual disempowerment risk, if we fleshed it out into a proper research field. (Analogously, ambitious mechinterp is a way to address both fast take-off and slow take-off risks.) And so I expect that a cluster forming around something like human empowerment would be more productive and less prone to capture.
avoiding using the term could be done mostly by either inventing another term for the the dynamic, or not thinking about the dynamic, or similar moves, which seem epistemically unhealthy
Yeah, “avoid using it altogether” would be too strong. Maybe something more like “I’ll avoid using it as a headline/pointer to a cluster of people/ideas, and only use it to describe the specific threat model”.
I mostly wouldn’t expect it to at this point, FWIW. The people engaged right now are by and large people sincerely grappling with the idea, and particularly people who are already bought into takeover risk. Whereas one of the main mechanisms by which I expect misuse of the idea is that people who are uncomfortable with the concept of “AI takeover” can still classify themselves as part of the AI safety coalition when it suits them.
As an illustration of this happening to Paul’s worldview, see this Vox article titled “AI disaster won’t look like the Terminator. It’ll be creepier.” My sense is that both Paul and Vox wanted to distance themselves from Eliezer’s scenarios, and so Paul phrased his scenario in a way which downplayed stuff like “robot armies” and then Vox misinterpreted Paul to further downplay that stuff. (More on this from Carl here.) Another example: Sam Altman has previously justified racing to AGI by appealing to the idea that a slow takeoff is better than a fast takeoff.
Now, some of these dynamics are unavoidable—we shouldn’t stop debating takeoffs just because people might misuse the concepts. But it’s worth keeping an eye out for ideas that are particularly prone to this, and gradual disempowerment seems like one.
Well, it’s much more convenient than “AI takeover”, and so the question is how much people are motivated to use it to displace the AI takeover meme in their internal narratives.
Kudos for doing so. I don’t mean to imply that you guys are unaware of this issue or negligent; IMO it’s a pretty hard problem to avoid. I agree that stuff like “understanding power” is nowhere near adequate as a replacement. However, I do think that there’s some concept like “empowering humans” which is a way to address both takeover risk and gradual disempowerment risk, if we fleshed it out into a proper research field. (Analogously, ambitious mechinterp is a way to address both fast take-off and slow take-off risks.) And so I expect that a cluster forming around something like human empowerment would be more productive and less prone to capture.
Yeah, “avoid using it altogether” would be too strong. Maybe something more like “I’ll avoid using it as a headline/pointer to a cluster of people/ideas, and only use it to describe the specific threat model”.