I agree that this risk is neglected. The entire field is pretty clearly severely neglected, but this perhaps more than other challenges surrounding AGI.
I differ from you in that I don’t regard this as an x-risk, but more like a mild s-risk. In many scenarios where someone decides to use their intent-aligned AGI to wipe out most of humanity and seize control of the future, they’ll have a vision of the future, and will (probably rapidly) rebuild civilization under that vision. It’s probably not the world I’d want, but it will probably preserve a lot of human values and either immediately or as the new god-tyrant appreciates the situation better, contain on net more joy than suffering. If we don’t get someone whose sadism-empathy balance is permanently negative. Which is quite possible.
Thus it’s a possible s-risk but unlikely to be a true x-risk for humanity—just for me and everyone I know. I love future humanity, too, so this isn’t a total loss from my or the utilitarian POV. But it’s still probably a large loss of future potential, a real catastrophe.
Thanks, Seth. What troubles me at the meta-level is the assumption of exclusive privilege implied by rationalist/utilitarian arguments, that of getting to make choices between extreme outcomes. It’s not just “I, as a rationalist, have considered the trade-offs between X and Y and, if forced to, will choose X.” It’s “I, a rationalist, believe that rationalism is superior to heuristic-based and otherwise inconsistent reasoning, and therefore assume the responsibility of making choices on behalf of those inferior reasoners.” There’s not much further to go to get to “I will conceal the ‘mild s-risk’ of the deaths of billions from them to get them to ally with me to avoid the x-risks that I am concerned about (but to which they, in their imperfect reasoning, are relatively indifferent).”
I agree that this risk is neglected. The entire field is pretty clearly severely neglected, but this perhaps more than other challenges surrounding AGI.
I have engaged with this issue in If we solve alignment, do we die anyway? and to a lesser degree in Whether governments will control AGI is important and neglected
I differ from you in that I don’t regard this as an x-risk, but more like a mild s-risk. In many scenarios where someone decides to use their intent-aligned AGI to wipe out most of humanity and seize control of the future, they’ll have a vision of the future, and will (probably rapidly) rebuild civilization under that vision. It’s probably not the world I’d want, but it will probably preserve a lot of human values and either immediately or as the new god-tyrant appreciates the situation better, contain on net more joy than suffering. If we don’t get someone whose sadism-empathy balance is permanently negative. Which is quite possible.
Thus it’s a possible s-risk but unlikely to be a true x-risk for humanity—just for me and everyone I know. I love future humanity, too, so this isn’t a total loss from my or the utilitarian POV. But it’s still probably a large loss of future potential, a real catastrophe.
Thanks, Seth. What troubles me at the meta-level is the assumption of exclusive privilege implied by rationalist/utilitarian arguments, that of getting to make choices between extreme outcomes. It’s not just “I, as a rationalist, have considered the trade-offs between X and Y and, if forced to, will choose X.” It’s “I, a rationalist, believe that rationalism is superior to heuristic-based and otherwise inconsistent reasoning, and therefore assume the responsibility of making choices on behalf of those inferior reasoners.” There’s not much further to go to get to “I will conceal the ‘mild s-risk’ of the deaths of billions from them to get them to ally with me to avoid the x-risks that I am concerned about (but to which they, in their imperfect reasoning, are relatively indifferent).”