But my moral code does include such statements as “you have no fundamental obligation to help other people.” I help people because I like to.
While I consider myself an altruist in principle (I have serious akrasia problems in practice), I do agree with this statement. Altruists don’t have any obligation to help people, it just often makes sense for them to do so; sometimes it doesn’t, and then the proper thing for them is not to do it.
Roko:
In the modern world, people have to make moral choices using their general intelligence, because there aren’t enough “yuck” and “yum” factors around to give guidance on every question. As such, we shouldn’t expect much more moral agreement from humans than from rational (or approximately rational) AIs.
There might not be enough “yuck” and “yum” factors around to offer direct guidance on every question, but they’re still the basis for abstract rational reasoning. Do you think “paperclip optimizer”-type AIs are impossible? If so, why? There’s nothing incoherent about a “maximize the number of paperclips over time” optimization criterion; if anything, it’s a lot simpler than those in use by humans.
Eliezer Yudkowsky:
If I have a value judgment that would not be interpersonally compelling to a supermajority of humankind even if they were fully informed, then it is proper for me to personally fight for and advocate that value judgment, but not proper for me to preemptively build an AI that enforces that value judgment upon the rest of humanity.
I don’t understand this at all. How is building a superintelligent AI not just a (highly effective, if you do it right) special method of personally fighting for your value judgement? Are you saying it’s ok to fight for it, as long as you don’t do it too effectively?
Jadagul:
While I consider myself an altruist in principle (I have serious akrasia problems in practice), I do agree with this statement. Altruists don’t have any obligation to help people, it just often makes sense for them to do so; sometimes it doesn’t, and then the proper thing for them is not to do it.Roko:
There might not be enough “yuck” and “yum” factors around to offer direct guidance on every question, but they’re still the basis for abstract rational reasoning. Do you think “paperclip optimizer”-type AIs are impossible? If so, why? There’s nothing incoherent about a “maximize the number of paperclips over time” optimization criterion; if anything, it’s a lot simpler than those in use by humans.Eliezer Yudkowsky:
I don’t understand this at all. How is building a superintelligent AI not just a (highly effective, if you do it right) special method of personally fighting for your value judgement? Are you saying it’s ok to fight for it, as long as you don’t do it too effectively?