I think what’s crucial here is that I think perfect alignment is very likely unattainable. If that’s true, then we’ll get some form of “value drift” in almost any realistic scenario. Over long periods, the world will start to look alien and inhuman. Here, the difficulty of alignment mostly sets how quickly this drift will occur, rather than determining whether the drift occurs at all.
Yep, and my disagreement as expressed in another comment is that I think that it’s not that hard to have robust corrigibility and there might also be a basin of corrigability.
The world looking alien isn’t necessarily a crux for me: it should be possible in principle to have AIs protect humans and do whatever is needed in the alien AI world while humans are sheltered and slowly self-enhance and pick successors (see the indirect normativity appendix in the ELK doc for some discussion of this sort of proposal).
I agree that perfect alignment will be hard, but I model the situation much more like a one time hair cut (at least in expectation) than exponential decay of control.
I expect that “humans stay in control via some indirect mechanism” (e.g. indirect normativity) or “humans coordinate to slow down AI progress at some point (possibly after solving all diseases and becoming wildly wealthy) (until some further point, e.g. human self-enhancement)” will both be more popular as proposals than the world you’re thinking about. Being popular isn’t sufficient: it also needs to be implementable and perhaps sufficiently legible, but I think at least implementable is likely.
Another mechanism that might be important is human self-enhancement: humans who care about staying in control can try to self-enhance to stay at least somewhat competitive with AIs while preserving their values. (This is not a crux for me and seems relatively marginal, but I thought I would mention it.)
Yep, and my disagreement as expressed in another comment is that I think that it’s not that hard to have robust corrigibility and there might also be a basin of corrigability.
The world looking alien isn’t necessarily a crux for me: it should be possible in principle to have AIs protect humans and do whatever is needed in the alien AI world while humans are sheltered and slowly self-enhance and pick successors (see the indirect normativity appendix in the ELK doc for some discussion of this sort of proposal).
I agree that perfect alignment will be hard, but I model the situation much more like a one time hair cut (at least in expectation) than exponential decay of control.
I expect that “humans stay in control via some indirect mechanism” (e.g. indirect normativity) or “humans coordinate to slow down AI progress at some point (possibly after solving all diseases and becoming wildly wealthy) (until some further point, e.g. human self-enhancement)” will both be more popular as proposals than the world you’re thinking about. Being popular isn’t sufficient: it also needs to be implementable and perhaps sufficiently legible, but I think at least implementable is likely.
Another mechanism that might be important is human self-enhancement: humans who care about staying in control can try to self-enhance to stay at least somewhat competitive with AIs while preserving their values. (This is not a crux for me and seems relatively marginal, but I thought I would mention it.)