Gotcha. Yes, it has a radically different meaning (and is unrelated to cipergoth’s use of the word in the comment you respond to). Also, “Friendly” is often capitalized when used in the sense you mean.
Roughly speaking, a nonhuman environment-optimizing system is said to be Friendly if the goals it is optimizing for are stable under reflection and self-modification (that is, no matter how sophisticated its awareness of what it is optimizing for, or its ability to change what it is optimizing for, becomes it will continue to optimize for those things), and if those goals are such that optimizing our environment for them results in an environment that’s optimal for humans.
Of course, defining that operationally is, as you say, non-trivial and of key importance. As far as I know that problem has not been solved yet. There’s a general consensus that even defining those goals (let alone designing and building a system that preserves them under reflection and self-modification) is a staggeringly hard problem.
Gotcha. Yes, it has a radically different meaning (and is unrelated to cipergoth’s use of the word in the comment you respond to). Also, “Friendly” is often capitalized when used in the sense you mean.
Roughly speaking, a nonhuman environment-optimizing system is said to be Friendly if the goals it is optimizing for are stable under reflection and self-modification (that is, no matter how sophisticated its awareness of what it is optimizing for, or its ability to change what it is optimizing for, becomes it will continue to optimize for those things), and if those goals are such that optimizing our environment for them results in an environment that’s optimal for humans.
Of course, defining that operationally is, as you say, non-trivial and of key importance. As far as I know that problem has not been solved yet. There’s a general consensus that even defining those goals (let alone designing and building a system that preserves them under reflection and self-modification) is a staggeringly hard problem.