that is the first time I have seen a definition of FAI. Is that the “official” definition
or just your own characterization?
Probably the closest thing I have seen from E.Y.:
“I use the term “Friendly AI” to refer to this whole challenge. Creating a mind that doesn’t kill people but does cure cancer …which is a rather limited way of putting it. More generally, the problem of pulling a mind out of mind design space, such that afterwards that you are glad you did it.”
This idea could be said to have some issues. An evil dictator pulling a mind out of mind design space, such that afterwards he is glad that he did it doesn’t seem much like quite what most of the world would regard as “friendly”. This definition is not very specific about exactly who the AI is “friendly” to.
Back in 2008 I asked “Friendly—to whom?” and got back this—though the reply now seems to have dropped out of the record.
Thanks for this link. Sounds kind of scary. American political conservatives will be thrilled. “I’m from the CEV and I’m here to help you.”
Incidentally, there should be an LW wiki entry for “CEV”. The acronym is thrown around a lot in the comments, but a definition is quite difficult to find. It would also be nice if there were a top-level posting on the topic to serve as an anchor-point for discussion. Because discussion is sorely needed.
It occurs to me that it would be very desirable to attempt to discover the CEV of humanity long before actually constructing an FAI to act under its direction. And I would be far more comfortable if the “E” stood for “expressed”, rather than “extrapolated”.
That, in fact, might be an attractive mission statement for an philanthropic foundation. Find the Coalesced/coherent Expressed/extrapolated Volition of mankind. Accomplish this by conducting opinion research, promoting responsible and enlightening debate and discussion, etc.
Speaking as an American, I certainly wish there were some serious financial support behind improving the quality of public policy debate, rather than behind supporting the agenda of one side in the debate or the other.
It occurs to me that it would be very desirable to attempt to discover the CEV of
humanity long before actually constructing an FAI to act under its direction.
Well, that brings us to a topic we have discussed before. Humans—like all other living systems—mosly act so as to increase entropy in their environment. That is http://originoflife.net/gods_utility_function/
CEV is a bizarre wishlist, apparently made with minimal consideration of implementation difficulties, and not paying too much attention to the order in which things are likely to play out.
I figure that—if the SIAI carries on down these lines—then they will be lumbered with a massively impractical design, and will be beaten to the punch by a long stretch—even if you ignore all their material about “provable correctness” and other safety features—which seem like more substantial handicaps to me.
CEV is a bizarre wishlist, apparently made with minimal consideration of implementation difficulties …
It is what the software professionals would call a preliminary requirements document. You are not supposed to worry about implementation difficulties at that stage of the process. Harsh reality will get its chance to force compromises later.
I think CEV is one proposal to consider, useful to focus discussion. I hate it, myself, and suspect that the majority of mankind would agree. I don’t want some machine that I have never met and don’t trust to be inferring my volition and acting on my behalf. The whole concept makes me want to go out and join some Luddite organization dedicated to making sure neither UFAI nor FAI ever happen. But, seen as an attempt to stimulate discussion, I think that the paper is great. And maybe discussion might improve the proposal enough to alleviate my concerns. Or discussion might show me that my concerns are baseless.
I sure hope EY isn’t deluded enough to think that initiatives like LW can be scaled up enough so as to improve the analytic capabilities of a sufficiently large fraction of mankind so that proposals like CEV will not encounter significant opposition.
It is what the software professionals would call a preliminary requirements
document. You are not supposed to worry about implementation difficulties at
that stage of the process. Harsh reality will get its chance to force compromises later.
What—not at all? You want the moon-onna-stick—so that goes into your “preliminary requirements” document?
Yes. Because there is always the possibility that some smart geek will say “‘moon-onna-stick’, huh? I bet I could do that. I see a clever trick.” Or maybe some other geek will say “Would you settle for Sputnik-on-a-stick?” and the User will say “Well, yes. Actually, that would be even better.”
At least that is what they preach in the Process books.
It sounds pretty surreal to me. I would usually favour some reality-imposed limits to fantasizing and wishful thinking from the beginning—unless there are practically no time constraints at all.
I sure hope EY isn’t deluded enough to think that initiatives like LW can be
scaled up enough so as to improve the analytic capabilities of a sufficiently
large fraction of mankind so that proposals like CEV will not encounter
significant opposition.
If there was ever any real chance of success, governments would be likely to object. Since they already have power, they are not going to want a bunch of geeks in a basement taking over the world with their intelligent machine—and redistributing all their assets for them.
Probably the closest thing I have seen from E.Y.:
“I use the term “Friendly AI” to refer to this whole challenge. Creating a mind that doesn’t kill people but does cure cancer …which is a rather limited way of putting it. More generally, the problem of pulling a mind out of mind design space, such that afterwards that you are glad you did it.”
http://singinst.org/media/thehumanimportanceoftheintelligenceexplosion
(29 minutes in)
This idea could be said to have some issues. An evil dictator pulling a mind out of mind design space, such that afterwards he is glad that he did it doesn’t seem much like quite what most of the world would regard as “friendly”. This definition is not very specific about exactly who the AI is “friendly” to.
Back in 2008 I asked “Friendly—to whom?” and got back this—though the reply now seems to have dropped out of the record.
There’s also another definition here.
Thanks for this link. Sounds kind of scary. American political conservatives will be thrilled. “I’m from the CEV and I’m here to help you.”
Incidentally, there should be an LW wiki entry for “CEV”. The acronym is thrown around a lot in the comments, but a definition is quite difficult to find. It would also be nice if there were a top-level posting on the topic to serve as an anchor-point for discussion. Because discussion is sorely needed.
It occurs to me that it would be very desirable to attempt to discover the CEV of humanity long before actually constructing an FAI to act under its direction. And I would be far more comfortable if the “E” stood for “expressed”, rather than “extrapolated”.
That, in fact, might be an attractive mission statement for an philanthropic foundation. Find the Coalesced/coherent Expressed/extrapolated Volition of mankind. Accomplish this by conducting opinion research, promoting responsible and enlightening debate and discussion, etc.
Speaking as an American, I certainly wish there were some serious financial support behind improving the quality of public policy debate, rather than behind supporting the agenda of one side in the debate or the other.
Well, that brings us to a topic we have discussed before. Humans—like all other living systems—mosly act so as to increase entropy in their environment. That is http://originoflife.net/gods_utility_function/
CEV is a bizarre wishlist, apparently made with minimal consideration of implementation difficulties, and not paying too much attention to the order in which things are likely to play out.
I figure that—if the SIAI carries on down these lines—then they will be lumbered with a massively impractical design, and will be beaten to the punch by a long stretch—even if you ignore all their material about “provable correctness” and other safety features—which seem like more substantial handicaps to me.
It is what the software professionals would call a preliminary requirements document. You are not supposed to worry about implementation difficulties at that stage of the process. Harsh reality will get its chance to force compromises later.
I think CEV is one proposal to consider, useful to focus discussion. I hate it, myself, and suspect that the majority of mankind would agree. I don’t want some machine that I have never met and don’t trust to be inferring my volition and acting on my behalf. The whole concept makes me want to go out and join some Luddite organization dedicated to making sure neither UFAI nor FAI ever happen. But, seen as an attempt to stimulate discussion, I think that the paper is great. And maybe discussion might improve the proposal enough to alleviate my concerns. Or discussion might show me that my concerns are baseless.
I sure hope EY isn’t deluded enough to think that initiatives like LW can be scaled up enough so as to improve the analytic capabilities of a sufficiently large fraction of mankind so that proposals like CEV will not encounter significant opposition.
That seems unlikely to help. Luddites have never had any power. Becoming a Luddite usually just makes you more xxxxxd.
What—not at all? You want the moon-onna-stick—so that goes into your “preliminary requirements” document?
Yes. Because there is always the possibility that some smart geek will say “‘moon-onna-stick’, huh? I bet I could do that. I see a clever trick.” Or maybe some other geek will say “Would you settle for Sputnik-on-a-stick?” and the User will say “Well, yes. Actually, that would be even better.”
At least that is what they preach in the Process books.
It sounds pretty surreal to me. I would usually favour some reality-imposed limits to fantasizing and wishful thinking from the beginning—unless there are practically no time constraints at all.
If there was ever any real chance of success, governments would be likely to object. Since they already have power, they are not going to want a bunch of geeks in a basement taking over the world with their intelligent machine—and redistributing all their assets for them.