The only part I disagree with strongly is the language of the last point. Referring to CEV as “THE morally right thing to do” makes it seem as if it were set in stone as the guaranteed best path to creating FAI, which it isn’t. EY argues that building Friendly AI instead of just letting the chips fall where they may is the morally right thing to do, and I’d agree with that, but not that CEV specifically is the right thing to do.
One general goal point for FAI is to target outcomes “at least as good” as those which would be caused by benevolent human mind upload(s). So, the kind of “moral development” that a community of uploads would undergo should be encapsulated within a FAI. In fact, any beneficial area of the moral state space that would be accessible starting from humans or any combination of humans and tools should be accessible by a good FAI design. CEV is one such proposal towards such a design.
As I understand it, yes, the thinking is to optimize for our terminal values instead of this hypothetical alien species or some compromise of the two. However, if values among different intelligent species converge given greater intelligence, knowledge, and self-reflection, then we would expect our FAI to have goals that converge with the alien FAI. If values do not converge, then we would suppose our FAI to have different values than alien FAIs.
A “terminal value” might include carefully thinking through philosophical questions such as this and designing the best goal content possible given these considerations. So, if there are hypothetical alien values that seem “correct” (or simply sufficiently desirable from the subjective perspective) to extrapolated humanity, these values would be integrated into the CEV-output.
I agree that EY does not assert that his proposed process for defining FAI’s optimization target (that is, seed AI calculating CEV) is necessarily the best path to FAI, nor that that proposed process is particularly right. Correction accepted.
And yes, I agree that on EY’s account, given an alien species whose values converge with ours, a system that optimizes for our terminal values also optimizes for theirs.
The only part I disagree with strongly is the language of the last point. Referring to CEV as “THE morally right thing to do” makes it seem as if it were set in stone as the guaranteed best path to creating FAI, which it isn’t. EY argues that building Friendly AI instead of just letting the chips fall where they may is the morally right thing to do, and I’d agree with that, but not that CEV specifically is the right thing to do.
One general goal point for FAI is to target outcomes “at least as good” as those which would be caused by benevolent human mind upload(s). So, the kind of “moral development” that a community of uploads would undergo should be encapsulated within a FAI. In fact, any beneficial area of the moral state space that would be accessible starting from humans or any combination of humans and tools should be accessible by a good FAI design. CEV is one such proposal towards such a design.
As I understand it, yes, the thinking is to optimize for our terminal values instead of this hypothetical alien species or some compromise of the two. However, if values among different intelligent species converge given greater intelligence, knowledge, and self-reflection, then we would expect our FAI to have goals that converge with the alien FAI. If values do not converge, then we would suppose our FAI to have different values than alien FAIs.
A “terminal value” might include carefully thinking through philosophical questions such as this and designing the best goal content possible given these considerations. So, if there are hypothetical alien values that seem “correct” (or simply sufficiently desirable from the subjective perspective) to extrapolated humanity, these values would be integrated into the CEV-output.
I agree that EY does not assert that his proposed process for defining FAI’s optimization target (that is, seed AI calculating CEV) is necessarily the best path to FAI, nor that that proposed process is particularly right. Correction accepted.
And yes, I agree that on EY’s account, given an alien species whose values converge with ours, a system that optimizes for our terminal values also optimizes for theirs.
Thanks.