I would offer that any set of goals given to this AGI would include the safety-concerns of humans. (Is this controversial?)
If anyone figures out how to give an AGI this goal, that would mean they know how to express the entire complex set of everything humans value, and express it with great precision in the form of mathematics/code without the use of any natural language words at all. No one on Earth knows how to do this for even a single human value. No one knows how to program an AI with a goal anywhere near that complex even if we did.
The relevant goal isn’t, “Make paperclips, but consistent with human values.”
It’s more like “Maximize this variable (which the programmer privately hopes corresponds to number of paperclips made), while simultaneously satisfying this set of constraints that might be terabytes in size or more because it corresponds to the entire axiology of all mankind including precise mathematical resolutions to all the moral disagreements we’ve been arguing about since at least the dawn of writing. Also, preserve that constraint through all possible future upgrades and changes that you make to yourself, or that humans try to make to you, unless the constraint itself indicates that it would be somehow better satisfied by letting the humans make the change.”
Strictly speaking about superhuman AGI: I believe you summarize the relative difficulty / impossibility of this task :) I can’t say I agree that the goal is void of human-values though (I’m talking about safety in particular—not sure if that’s make a difference?) --seems impractical right from the start?
I also think these considerations seem manageable though, when considering the narrow AI that we are producing as of today. But where’s the appetite to continue on the ANI road? I can’t really believe we wouldn’t want more of the same, in different fields of endeavor…
If anyone figures out how to give an AGI this goal, that would mean they know how to express the entire complex set of everything humans value, and express it with great precision in the form of mathematics/code without the use of any natural language words at all. No one on Earth knows how to do this for even a single human value. No one knows how to program an AI with a goal anywhere near that complex even if we did.
The relevant goal isn’t, “Make paperclips, but consistent with human values.”
It’s more like “Maximize this variable (which the programmer privately hopes corresponds to number of paperclips made), while simultaneously satisfying this set of constraints that might be terabytes in size or more because it corresponds to the entire axiology of all mankind including precise mathematical resolutions to all the moral disagreements we’ve been arguing about since at least the dawn of writing. Also, preserve that constraint through all possible future upgrades and changes that you make to yourself, or that humans try to make to you, unless the constraint itself indicates that it would be somehow better satisfied by letting the humans make the change.”
Strictly speaking about superhuman AGI: I believe you summarize the relative difficulty / impossibility of this task :) I can’t say I agree that the goal is void of human-values though (I’m talking about safety in particular—not sure if that’s make a difference?) --seems impractical right from the start?
I also think these considerations seem manageable though, when considering the narrow AI that we are producing as of today. But where’s the appetite to continue on the ANI road? I can’t really believe we wouldn’t want more of the same, in different fields of endeavor…