Oh, come on, Eliezer, of course you thought of it. ;) However, it might not have been something that bothered you, as in-
A) You didn’t believe actually having autonomy mattered as long as people feel like they do (ie a Matrix/Nexus situation). I have heard this argued. Would it matter to you if you found out your whole life was a simulation? Some say no. I say yes. Matter of taste perhaps?
B) OR You find it self evident that ‘real’ autonomy would be extrapolated by the AI as something essential to human happiness, such that an intelligence observing people and maximizing our utility wouldn’t need to be told ‘allow autonomy.’ This I would disagree with.
C) OR You recognize that this is a problem with a non-obvious solution to an AI, and thus intend to deal with it somehow in code ahead of time, before starting the volition extrapolating AI. Your response indicates you feel this way. However, I am concerned even beyond setting an axiomatic function for ‘allow autonomy’ in a program. There are probably an infinite number of ways that an AI can find ways to carry out its stated function that will somehow ‘game’ our own system and lead to suboptimal or outright repugnant results (ie everyone being trapped in a permanent quest- maybe the AI avoids the problem of ‘it has to be real’ by actually creating a magic ring that needs to be thrown into a volcano every 6 years or so). You don’t need me telling you that! Maximizing utility while deluding us about reality is only one. It seems impossible that we could axiomatically safeguard against all possibilities. Assimov was a pretty smart cookie, and his ‘3 laws’ are certainly not sufficient. ‘Eliezer’s million lines of code’ might cover a much larger range of AI failures, but how could you ever be sure? The whole project just seems insanely dangerous. Or are you going to address safety concerns in another post in this series?
Oh, come on, Eliezer, of course you thought of it. ;) However, it might not have been something that bothered you, as in- A) You didn’t believe actually having autonomy mattered as long as people feel like they do (ie a Matrix/Nexus situation). I have heard this argued. Would it matter to you if you found out your whole life was a simulation? Some say no. I say yes. Matter of taste perhaps?
B) OR You find it self evident that ‘real’ autonomy would be extrapolated by the AI as something essential to human happiness, such that an intelligence observing people and maximizing our utility wouldn’t need to be told ‘allow autonomy.’ This I would disagree with.
C) OR You recognize that this is a problem with a non-obvious solution to an AI, and thus intend to deal with it somehow in code ahead of time, before starting the volition extrapolating AI. Your response indicates you feel this way. However, I am concerned even beyond setting an axiomatic function for ‘allow autonomy’ in a program. There are probably an infinite number of ways that an AI can find ways to carry out its stated function that will somehow ‘game’ our own system and lead to suboptimal or outright repugnant results (ie everyone being trapped in a permanent quest- maybe the AI avoids the problem of ‘it has to be real’ by actually creating a magic ring that needs to be thrown into a volcano every 6 years or so). You don’t need me telling you that! Maximizing utility while deluding us about reality is only one. It seems impossible that we could axiomatically safeguard against all possibilities. Assimov was a pretty smart cookie, and his ‘3 laws’ are certainly not sufficient. ‘Eliezer’s million lines of code’ might cover a much larger range of AI failures, but how could you ever be sure? The whole project just seems insanely dangerous. Or are you going to address safety concerns in another post in this series?