Then they don’t know the true difference between the two possible lives, do they?
“True difference” gets me thinking of “no true Scotsman”. Has there ever been anybody who truly knew the difference between two possible lives? Even if someone could be reincarnated and retain memories the order would likely alter their perceptions.
I’m very interested in how Eliezer gets from his meta-ethics to utilitarianism
He’s not a strict utilitarian in the “happiness alone” sense. He has an aversion to wireheading, which maximizes the classic version of utility.
I know you frown upon mentioning evolutionary psychology, but is it really a huge stretch to surmise that the more even-keeled, loving and peaceful tribes of our ancestors would out-survive the wilder warmongers who killed each other out?
Yes, it is. The peaceful ones would be vulnerable to being wiped out by the more warlike ones. Or, more accurately (group selection isn’t as big a factor given intergroup variance being smaller than intragroup variance), the members of the peaceful tribe more prone to violence would achieve dominance as hawks among doves do. Among the Yanonamo we find high reproductive success among men who have killed. The higher the bodycount, the more children. War and murder appear to be human universals.
Eliezer’s obvious awareness of rationalization is encouraging
Awareness of biases can increase errors, so it’s not encouraging enough given the stakes.
Finally, I would think there would be more than one AI programmer, reducing the risk of deliberate evil
I’m not really worried about that. No one is a villain in their own story, and people we would consider deviants would likely be filtered out of the Institute and would probably be attracted to other career paths anyway. The problem exists, but I’m more concerned with well-meaning designers creating something that goes off in directions we can’t anticipate.
Caledonian, Eliezer never said anything about not bothering to look for arguments. His idea is to find out how he found respond if he were confronted with all arguments. He seems to assume that he (or the simulation of him) will correctly evaluate arguments. His point about no universal arguments is that he has to start with himself rather than some ghostly ideal behind a veil of ignorance or something like that.
Then they don’t know the true difference between the two possible lives, do they?
“True difference” gets me thinking of “no true Scotsman”. Has there ever been anybody who truly knew the difference between two possible lives? Even if someone could be reincarnated and retain memories the order would likely alter their perceptions.
I’m very interested in how Eliezer gets from his meta-ethics to utilitarianism
He’s not a strict utilitarian in the “happiness alone” sense. He has an aversion to wireheading, which maximizes the classic version of utility.
I know you frown upon mentioning evolutionary psychology, but is it really a huge stretch to surmise that the more even-keeled, loving and peaceful tribes of our ancestors would out-survive the wilder warmongers who killed each other out?
Yes, it is. The peaceful ones would be vulnerable to being wiped out by the more warlike ones. Or, more accurately (group selection isn’t as big a factor given intergroup variance being smaller than intragroup variance), the members of the peaceful tribe more prone to violence would achieve dominance as hawks among doves do. Among the Yanonamo we find high reproductive success among men who have killed. The higher the bodycount, the more children. War and murder appear to be human universals.
Eliezer’s obvious awareness of rationalization is encouraging
Awareness of biases can increase errors, so it’s not encouraging enough given the stakes.
Finally, I would think there would be more than one AI programmer, reducing the risk of deliberate evil
I’m not really worried about that. No one is a villain in their own story, and people we would consider deviants would likely be filtered out of the Institute and would probably be attracted to other career paths anyway. The problem exists, but I’m more concerned with well-meaning designers creating something that goes off in directions we can’t anticipate.
Caledonian, Eliezer never said anything about not bothering to look for arguments. His idea is to find out how he found respond if he were confronted with all arguments. He seems to assume that he (or the simulation of him) will correctly evaluate arguments. His point about no universal arguments is that he has to start with himself rather than some ghostly ideal behind a veil of ignorance or something like that.