Interesting quote from Eliezer’s 1996 essay Staring into the Singularity, from the section about mind-uploading:
Maybe, after Transcending, you will perhaps be a bit changed. If that is so, it is both inevitable and morally right. Given that absolutely nothing you can do will change that, why worry? Save your anxiety for what you can affect.
(The same goes for worries about hostile superintelligences wiping out humanity. I cannot guarantee this won’t happen, since I’m not superintelligent. However, I am reasonably (95%) certain that the Powers will be ethical; I just don’t know what really is “ethical”, the true moral right. If wiping out your creators is morally wrong, it won’t happen. If we do get wiped out, it will be because, were we upgraded to superintelligences instead, we would see the moral necessity and commit suicide. If that’s the case, there is absolutely nothing we can do about it, both in the long run (see above) and the short run (see below); and, even were we granted the opportunity, it would be morally wrong to take it. Observer-dependent morality is a chimerical artifact of evolutionary competition; it dissolves under the application of sufficient intelligence. If you think it would be wrong to kill you, and you are rationally correct about it, presumably the Powers will see the same argument and you’ll be perfectly safe. Again, why worry?)
I knew he used to believe this, since he mentioned it in My Childhood Death Spiral, but it’s funny seeing Eliezer Yudkowsky argue that he’s 95% certain ASI will be ethical. Not trying to dunk on Eliezer in particular; it makes me worry that anyone could make giant foundational errors that completely reverse the sign of their impact.
Interesting quote from Eliezer’s 1996 essay Staring into the Singularity, from the section about mind-uploading:
I knew he used to believe this, since he mentioned it in My Childhood Death Spiral, but it’s funny seeing Eliezer Yudkowsky argue that he’s 95% certain ASI will be ethical. Not trying to dunk on Eliezer in particular; it makes me worry that anyone could make giant foundational errors that completely reverse the sign of their impact.
It seems that where Eliezer was in 1996, many people are today.