Thanks for writing this. I think this topic is generally a blind spot for LessWrong users, and it’s kind of embarrassing how little thought this community (myself included) has given to the question of whether a typical future with human control over AI is good.
(This actually slightly broadens the question, compared to yours. Because you talk about “a human” taking over the world with AGI, and make guesses about the personality of such a human after conditioning on them deciding to do that. But I’m not even confident that AGI-enabled control of the world by e.g. the US government would be good.)
Concretely, I think that a common perspective people take is: “What would it take for the future to go really really well, by my lights”, and the answer to that question probably involves human control of AGI. But that’s not really the action-relevant question. The action-relevant question, for deciding whether you want to try to solve alignment, is how the average world with human-controlled AGI compares to the average AGI-controlled world. And… I don’t know, in part for the reasons you suggest.
Thanks for writing this. I think this topic is generally a blind spot for LessWrong users, and it’s kind of embarrassing how little thought this community (myself included) has given to the question of whether a typical future with human control over AI is good.
I don’t think it’s embarrassing or a blind spot. I think I agree that it should receive more thought on the margin, and I of course agree that it should receive more thought all things considered. There’s a lot to think about! You may be underestimating how much thought has been devoted to this so far. E.g. it was a common topic of discussion at the center on long-term-risk while I was there. And it’s not like LW didn’t consider the question until now; my recollection is that various of us considered it & concluded that yeah probably human takeover is better than AI takeover in expectation for the reasons discussed in this post.
Side note: The title of this post is “Human Takeover Might Be Worse than AI Takeover” but people seem to be reading it as “Human Takeover Will Be Worse In Expectation than AI Takeover” and when I actually read the text I come away thinking “OK yeah, these arguments make me think that human takeover will be better in expectation than AI takeover, but with some significant uncertainty.”
The action-relevant question, for deciding whether you want to try to solve alignment, is how the average world with human-controlled AGI compares to the average AGI-controlled world.
To nitpick a little, it’s more like “the average world where we just barely didn’t solve alignment, versus the average world where we just barely did” (to the extent making things binary in this way is sensible), which I think does affect the calculus a little—marginal AGI-controlled worlds are more likely to have AIs which maintain some human values.
(Though one might be able to work on alignment in order to improve the quality of AGI-controlled worlds from worse to better ones, which mitigates this effect.)
Thanks for writing this. I think this topic is generally a blind spot for LessWrong users, and it’s kind of embarrassing how little thought this community (myself included) has given to the question of whether a typical future with human control over AI is good.
(This actually slightly broadens the question, compared to yours. Because you talk about “a human” taking over the world with AGI, and make guesses about the personality of such a human after conditioning on them deciding to do that. But I’m not even confident that AGI-enabled control of the world by e.g. the US government would be good.)
Concretely, I think that a common perspective people take is: “What would it take for the future to go really really well, by my lights”, and the answer to that question probably involves human control of AGI. But that’s not really the action-relevant question. The action-relevant question, for deciding whether you want to try to solve alignment, is how the average world with human-controlled AGI compares to the average AGI-controlled world. And… I don’t know, in part for the reasons you suggest.
I don’t think it’s embarrassing or a blind spot. I think I agree that it should receive more thought on the margin, and I of course agree that it should receive more thought all things considered. There’s a lot to think about! You may be underestimating how much thought has been devoted to this so far. E.g. it was a common topic of discussion at the center on long-term-risk while I was there. And it’s not like LW didn’t consider the question until now; my recollection is that various of us considered it & concluded that yeah probably human takeover is better than AI takeover in expectation for the reasons discussed in this post.
Side note: The title of this post is “Human Takeover Might Be Worse than AI Takeover” but people seem to be reading it as “Human Takeover Will Be Worse In Expectation than AI Takeover” and when I actually read the text I come away thinking “OK yeah, these arguments make me think that human takeover will be better in expectation than AI takeover, but with some significant uncertainty.”
To nitpick a little, it’s more like “the average world where we just barely didn’t solve alignment, versus the average world where we just barely did” (to the extent making things binary in this way is sensible), which I think does affect the calculus a little—marginal AGI-controlled worlds are more likely to have AIs which maintain some human values.
(Though one might be able to work on alignment in order to improve the quality of AGI-controlled worlds from worse to better ones, which mitigates this effect.)
See also: “Which World Gets Saved”