That was not my point. I was not appealing to some abstract “friendly AI in the sky” as a new kind of rationalist god.
My post was meant to address the people in the forum, a sort of warning that shouted: “Hey, lesswrongians! I saw you are in love with complexity and so on, but don’t forget that people might have radically different preferences. Indeed, these are mine. Don’t forget to include them in a friendly AI, if you happen to build one”.
Baseline: I think the fun theory that circulates the most in this forum, as per Yudkowsky sequence, is dangerously narrow. That could have potentially catastrophic consequences, and I wanted to prevent that.
The most commonly discussed FAI won’t have any (human) terminal values built in (except maybe for bootstrapping), but will examine humans to get a complete understanding of what humans actually value and then optimize that.
Therefore, whatever we humans now think about what might be fun or not is fairly irrelevant for the FAI simply because we don’t know what our values are. (That is also why I referenced the confusion between terminal and instrumental values in my first comment. I think that pretty much everyone talking about their “values” is only talking about instrumental values and that properly reduced terminal values are much, much simpler.)
That was not my point. I was not appealing to some abstract “friendly AI in the sky” as a new kind of rationalist god.
My post was meant to address the people in the forum, a sort of warning that shouted: “Hey, lesswrongians! I saw you are in love with complexity and so on, but don’t forget that people might have radically different preferences. Indeed, these are mine. Don’t forget to include them in a friendly AI, if you happen to build one”.
Baseline: I think the fun theory that circulates the most in this forum, as per Yudkowsky sequence, is dangerously narrow. That could have potentially catastrophic consequences, and I wanted to prevent that.
The most commonly discussed FAI won’t have any (human) terminal values built in (except maybe for bootstrapping), but will examine humans to get a complete understanding of what humans actually value and then optimize that.
Therefore, whatever we humans now think about what might be fun or not is fairly irrelevant for the FAI simply because we don’t know what our values are. (That is also why I referenced the confusion between terminal and instrumental values in my first comment. I think that pretty much everyone talking about their “values” is only talking about instrumental values and that properly reduced terminal values are much, much simpler.)