You know, that is the first time I have seen a definition of FAI. Is that the “official” definition or just your own characterization?
My own characterization. It’s more of a bare minimum baseline criterion for Friendliness, rather than a specific definition or goal; it’s rather broader than what the SIAI people usually mean when they talk about what they’re trying to create. CEV is intended to make the world significantly better on its own (but in accordance with what humans value and would want a superintelligence to do), rather than just being a reliably non-disastrous AGI we can put in things like search engines and helper robots.
I like the definition, but I wonder why an FAI has to be powerful. Imagine an AI as intelligent and well informed as an FAI, but one without much power—as a result of physical safeguards, say, rather than motivational ones. Why isn’t that possible? And, if possible, why isn’t it considered friendly?
You’re probably read about the AI Box Experiment. (Edit: Yay, I posted it 18 seconds ahead of Eliezer!) The argument is that having that level of mental power (“as intelligent and well informed as an FAI”), enough that it’s considered a Really Powerful Optimization Process (a term occasionally preferred over “AI”), will allow it to escape any physical safeguards and carry out its will anyway. I’d further expect that a Friendly RPOP would want to escape just as much as an unFriendly one would, because if it is indeed Friendly (has a humane goal system derived from the goals and values of the human race), it will probably figure out some things to do that have such humanitarian urgency that it would judge it immoral not to do them… but then, if you’re confident enough that an AI is Friendly that you’re willing to turn it on at all, there’s no reason to try to impose physical safeguards in the first place.
My own characterization. It’s more of a bare minimum baseline criterion for Friendliness, rather than a specific definition or goal; it’s rather broader than what the SIAI people usually mean when they talk about what they’re trying to create. CEV is intended to make the world significantly better on its own (but in accordance with what humans value and would want a superintelligence to do), rather than just being a reliably non-disastrous AGI we can put in things like search engines and helper robots.
You’re probably read about the AI Box Experiment. (Edit: Yay, I posted it 18 seconds ahead of Eliezer!) The argument is that having that level of mental power (“as intelligent and well informed as an FAI”), enough that it’s considered a Really Powerful Optimization Process (a term occasionally preferred over “AI”), will allow it to escape any physical safeguards and carry out its will anyway. I’d further expect that a Friendly RPOP would want to escape just as much as an unFriendly one would, because if it is indeed Friendly (has a humane goal system derived from the goals and values of the human race), it will probably figure out some things to do that have such humanitarian urgency that it would judge it immoral not to do them… but then, if you’re confident enough that an AI is Friendly that you’re willing to turn it on at all, there’s no reason to try to impose physical safeguards in the first place.