I think there are multiple legitimate ways that someone’s values could evolve, but some ways are illegitimate. A reflection process should probably reject slavery and avoid joining cults, but maybe it doesn’t matter which exact level of libertarianism it suggests.
People mean something when they talk about “ethics” and “true values,” even if there’s no objective truth of the matter. I’m talking about whatever it is they mean.
Vladimir Nesov has a suggestion here about how this could be done[1]. I don’t think it quite works, but to the extent that it is effective, it can be extended beyond just the influence of superintelligence to other types of new territory (and superintelligence as well, since Nesov’s proposal requires a Sysop[2], though presumably with a lot of transhumanist 3+1- or 4-volume locked out by Nesov’s design).
I think there are multiple legitimate ways that someone’s values could evolve, but some ways are illegitimate. A reflection process should probably reject slavery and avoid joining cults, but maybe it doesn’t matter which exact level of libertarianism it suggests.
People mean something when they talk about “ethics” and “true values,” even if there’s no objective truth of the matter. I’m talking about whatever it is they mean.
Vladimir Nesov has a suggestion here about how this could be done[1]. I don’t think it quite works, but to the extent that it is effective, it can be extended beyond just the influence of superintelligence to other types of new territory (and superintelligence as well, since Nesov’s proposal requires a Sysop[2], though presumably with a lot of transhumanist 3+1- or 4-volume locked out by Nesov’s design).
https://www.lesswrong.com/posts/vzHtHHBJoKATi5SeK/empowerment-corrigibility-etc-are-simple-abstractions-of-a?commentId=BjQrqeKfov946oAKj
Creating Friendly AI 1.0 by Eliezer Yudkowsky (2000), Section 5.9.2