I liked this post, but I think there’s a good chance that the future doesn’t end up looking like a central example of either “a single human seizes power” or “a single rogue AI seizes power”. Some other possible futures:
Control over the future by a group of humans, like “the US government” or “the shareholders of an AI lab” or “direct democracy over all humans who existed in 2029”
Takeover via an AI that a specific human crafted to do a good job at enacting that human’s values in particular, but which the human has no further steering power over
Lots of different actors (both human and AI) respecting one another’s property rights and pursuing goals within negotiated regions of spacetime, with no one actor having power over the majority of available resources
A future in which one human has extremely large amounts of power, but they acquired that power via trade and consensual agreements through their immense (ASI-derived) material wealth rather than via the sorts of coercive actions we tend to imagine with words like “takeover”.
A singleton ASI is in decisive control of the future, and among its values are a strong commitment to listen to human input and behave according to its understanding of collective human preferences, though maybe not its single overriding concern.
I’d be pretty excited to see more attempts at comparing these kinds of scenarios for plausibility and for how well the world might go conditional on their occurrence.
(I think it’s fairly likely that lots of these scenarios will eventually converge on something like the standard picture of one relatively coherent nonhuman agent doing vaguely consequentialist maximization across the universe, after sufficient negotiation and value-reflection and so on, but you might still care quite a lot about how the initial conditions shake out, and the dumbest AI capable of performing a takeover is probably very far from that limiting state.)
I liked this post, but I think there’s a good chance that the future doesn’t end up looking like a central example of either “a single human seizes power” or “a single rogue AI seizes power”. Some other possible futures:
Control over the future by a group of humans, like “the US government” or “the shareholders of an AI lab” or “direct democracy over all humans who existed in 2029”
Takeover via an AI that a specific human crafted to do a good job at enacting that human’s values in particular, but which the human has no further steering power over
Lots of different actors (both human and AI) respecting one another’s property rights and pursuing goals within negotiated regions of spacetime, with no one actor having power over the majority of available resources
A governance structure which nominally leaves particular humans in charge, and which the AIs involved are rule-abiding enough to respect, but in which things are sufficiently complicated and beyond human understanding that most decisions lack meaningful human oversight.
A future in which one human has extremely large amounts of power, but they acquired that power via trade and consensual agreements through their immense (ASI-derived) material wealth rather than via the sorts of coercive actions we tend to imagine with words like “takeover”.
A singleton ASI is in decisive control of the future, and among its values are a strong commitment to listen to human input and behave according to its understanding of collective human preferences, though maybe not its single overriding concern.
I’d be pretty excited to see more attempts at comparing these kinds of scenarios for plausibility and for how well the world might go conditional on their occurrence.
(I think it’s fairly likely that lots of these scenarios will eventually converge on something like the standard picture of one relatively coherent nonhuman agent doing vaguely consequentialist maximization across the universe, after sufficient negotiation and value-reflection and so on, but you might still care quite a lot about how the initial conditions shake out, and the dumbest AI capable of performing a takeover is probably very far from that limiting state.)