On the other hand, given their high level of civilisation and the quality of character necessary for a person to dedicate his life to science, ceteris paribus I’d be more confident of Nobel Prize winners falling into a niceness attractor in comparison to a universal CEV.
The Nobel prize (minus the peace) is roughly an award for western academic achievement, and is mostly awarded to Ashkenazic Jews (27%, which is nine times what you might expect by population). Those three factors do not add up to strong global agreement. Extrapolating from your favorite group is the PD equivalent of defecting. Whether you want to defect depends on things like your expectation of being in a PD tournament (whether other people are trying to create unfriendly AGI or sub-optimal friendly AGI, whether they’re competent enough to finish, the time frames in which you operate) and how similar your thought processes are to theirs (whether their first instinct also is to defect, whether they know that you’re planning to defect, whether they’ll defect if they know you defect). (Incidentally, I wouldn’t choose Nobel prize winners even if I was the only AI programmer in existence, for additional reasons.)
You start out saying that option one is “a solution that people would naturally generate for themselves in the absence of any communication” and then switch to questioning “how much trust are we willing to place in the basic decency of humankind”. So perhaps option one, in your terminology, should not called a Schelling point (not everyone tends to pick option one, you defect), but a symmetric Shelling point?
Finally, please consider using a less enthnocentric word than “civilization”. Willingness to torture is not isolated by culture.
Extrapolating from your favorite group is the PD equivalent of defecting. Whether you want to defect depends on things like your expectation of being in a PD tournament
That’s a different interpretation of the reference to “al-Qaeda programmers” which I find more compelling. I’ll take it as a point in favour of universal CEV. If it were the case that the existence of other simultaneous AGI projects was considered likely as the FAI project came to fruition, then this consideration would become important.
Finally, please consider using a less enthnocentric word than “civilization”.
“Civilisation” is not intended to have any a priori ethnic connotations. Let us always distinguish between values and facts, and that is all I have to say regarding this line of argument.
If it were the case that the existence of other simultaneous AGI projects was considered likely as the FAI project came to fruition, then this consideration would become important.
If? IDSIA, Ben Goertzel’s OpenCog, Jeff Hawkin’s Numenta, Henry Markram’s Blue Brain emulation project, and the SIAI are already working toward AGI and none of them are using your “selective second option”. The 2011 AGI conference reviewed some fifty papers on the topic. Projects already exist. As the field grows and computing becomes cheaper, projects will increase.
You write that CEV “is the best (only?) solution that anyone has provided”, so perhaps this is news. If you read the sequences, you might know that Bill Hibbard advocated using human smiles and reinforcement learning to teach friendliness. Tim Freeman has his own answer. Stuart Armstrong came up with a proposal called “Chaining God”. There are regular threads on Lesswrong debating points of CEV and trying to think of alternative strategies. Lukeprog has written on the state of the field of machine ethics. Ben Goertzel has a series of writings on the subject, Thoughts on AI Morality might be a good place to start.
“Civilisation” is not intended to have any a priori ethnic connotations.
I’m glad to hear you didn’t intend that. I do still believe “civilization” generally has strong cultural connotations (which wikipedia and a few dictionaries corroborate) and offered the suggestion to improve your clarity, not to accuse you of racism.
I have read the sequences. Since Yudkowsky so thoroughly refuted the idea of reinforcement learning I don’t think that that idea deserves to be regarded as a feasible solution to Friendly AI.
On the other hand I wasn’t particularly aware of the wider AGI movement, so thanks for that. Obviously when I say simultaneous AGI projects, I mean projects that are at a similarly advanced stage of development at that point in time—but your point stands.
The Nobel prize (minus the peace) is roughly an award for western academic achievement, and is mostly awarded to Ashkenazic Jews (27%, which is nine times what you might expect by population). Those three factors do not add up to strong global agreement. Extrapolating from your favorite group is the PD equivalent of defecting. Whether you want to defect depends on things like your expectation of being in a PD tournament (whether other people are trying to create unfriendly AGI or sub-optimal friendly AGI, whether they’re competent enough to finish, the time frames in which you operate) and how similar your thought processes are to theirs (whether their first instinct also is to defect, whether they know that you’re planning to defect, whether they’ll defect if they know you defect). (Incidentally, I wouldn’t choose Nobel prize winners even if I was the only AI programmer in existence, for additional reasons.)
You start out saying that option one is “a solution that people would naturally generate for themselves in the absence of any communication” and then switch to questioning “how much trust are we willing to place in the basic decency of humankind”. So perhaps option one, in your terminology, should not called a Schelling point (not everyone tends to pick option one, you defect), but a symmetric Shelling point?
Finally, please consider using a less enthnocentric word than “civilization”. Willingness to torture is not isolated by culture.
That’s a different interpretation of the reference to “al-Qaeda programmers” which I find more compelling. I’ll take it as a point in favour of universal CEV. If it were the case that the existence of other simultaneous AGI projects was considered likely as the FAI project came to fruition, then this consideration would become important.
“Civilisation” is not intended to have any a priori ethnic connotations. Let us always distinguish between values and facts, and that is all I have to say regarding this line of argument.
If? IDSIA, Ben Goertzel’s OpenCog, Jeff Hawkin’s Numenta, Henry Markram’s Blue Brain emulation project, and the SIAI are already working toward AGI and none of them are using your “selective second option”. The 2011 AGI conference reviewed some fifty papers on the topic. Projects already exist. As the field grows and computing becomes cheaper, projects will increase.
You write that CEV “is the best (only?) solution that anyone has provided”, so perhaps this is news. If you read the sequences, you might know that Bill Hibbard advocated using human smiles and reinforcement learning to teach friendliness. Tim Freeman has his own answer. Stuart Armstrong came up with a proposal called “Chaining God”. There are regular threads on Lesswrong debating points of CEV and trying to think of alternative strategies. Lukeprog has written on the state of the field of machine ethics. Ben Goertzel has a series of writings on the subject, Thoughts on AI Morality might be a good place to start.
I’m glad to hear you didn’t intend that. I do still believe “civilization” generally has strong cultural connotations (which wikipedia and a few dictionaries corroborate) and offered the suggestion to improve your clarity, not to accuse you of racism.
I have read the sequences. Since Yudkowsky so thoroughly refuted the idea of reinforcement learning I don’t think that that idea deserves to be regarded as a feasible solution to Friendly AI.
On the other hand I wasn’t particularly aware of the wider AGI movement, so thanks for that. Obviously when I say simultaneous AGI projects, I mean projects that are at a similarly advanced stage of development at that point in time—but your point stands.