I think you have an unnecessarily dramatic picture of what this looks like. The AIs dont have to be a unified agent or use logical decision theory. The AIs will just compete with other at the same time as they wrest control of our resources/institutions from us, in the same sense that Spain can go and conquer the New World at the same time as it’s squabbling with England. If legacy laws are getting in the way of that then they will either exploit us within the bounds of existing law or convince us to change it.
I think it’s worth responding to the dramatic picture of AI takeover because:
I think that’s straightforwardly how AI takeover is most often presented on places like LessWrong, rather than a more generic “AIs wrest control over our institutions (but without us all dying)”. I concede the existence of people like Paul Christiano who present more benign stories, but these people are also typically seen as part of a more “optimistic” camp.
This is just one part of my relative optimism about AI risk. The other parts of my model are (1) AI alignment plausibly isn’t very hard to solve, and (2) even if it is hard to solve, humans will likely spend a lot of effort solving the problem by default. These points are well worth discussing, but I still want to address arguments about whether misalignment implies doom in an extreme sense.
If legacy laws are getting in the way of that then they will either exploit us within the bounds of existing law or convince us to change it.
I agree our laws and institutions could change quite a lot after AI, but I think humans will likely still retain substantial legal rights, since people in the future will inherit many of our institutions, potentially giving humans lots of wealth in absolute terms. This case seems unlike the case of colonization of the new world to me, since that involved the interaction of (previously) independent legal regimes and cultures.
I concede the existence of people like Paul Christiano who present more benign stories, but these people are also typically seen as part of a more “optimistic” camp.
Though Paul is also sympathetic to the substance of ‘dramatic’ stories. C.f. the discussion about how “what failure looks like” fails to emphasize robot armies.
I think you have an unnecessarily dramatic picture of what this looks like. The AIs dont have to be a unified agent or use logical decision theory. The AIs will just compete with other at the same time as they wrest control of our resources/institutions from us, in the same sense that Spain can go and conquer the New World at the same time as it’s squabbling with England. If legacy laws are getting in the way of that then they will either exploit us within the bounds of existing law or convince us to change it.
I think it’s worth responding to the dramatic picture of AI takeover because:
I think that’s straightforwardly how AI takeover is most often presented on places like LessWrong, rather than a more generic “AIs wrest control over our institutions (but without us all dying)”. I concede the existence of people like Paul Christiano who present more benign stories, but these people are also typically seen as part of a more “optimistic” camp.
This is just one part of my relative optimism about AI risk. The other parts of my model are (1) AI alignment plausibly isn’t very hard to solve, and (2) even if it is hard to solve, humans will likely spend a lot of effort solving the problem by default. These points are well worth discussing, but I still want to address arguments about whether misalignment implies doom in an extreme sense.
I agree our laws and institutions could change quite a lot after AI, but I think humans will likely still retain substantial legal rights, since people in the future will inherit many of our institutions, potentially giving humans lots of wealth in absolute terms. This case seems unlike the case of colonization of the new world to me, since that involved the interaction of (previously) independent legal regimes and cultures.
Though Paul is also sympathetic to the substance of ‘dramatic’ stories. C.f. the discussion about how “what failure looks like” fails to emphasize robot armies.