What Robin is saying is, there’s a difference between
“metrics that correlate well enough with what you really want that you can make them the subject of contracts with other human beings”, and
“metrics that correlate well enough with what you really want that you can make them the subject of a transhuman intelligence’s goals”.
There are creative avenues of fulfilling the letter without fulfilling the spirit that would never occur to you but would almost certainly occur to a superintelligence, not because xe is malicious, but because they’re the optimal way to achieve the explicit goal set for xer. Your optimism, your belief that you can easily specify a goal (in computer code, not even English words) which admits of no undesirable creative shortcuts, is grossly misplaced once you bring smarter-than-human agents into the discussion. You cannot patch this problem; it has to be rigorously solved, or your AI wrecks the world.
What Robin is saying is, there’s a difference between
“metrics that correlate well enough with what you really want that you can make them the subject of contracts with other human beings”, and
“metrics that correlate well enough with what you really want that you can make them the subject of a transhuman intelligence’s goals”.
There are creative avenues of fulfilling the letter without fulfilling the spirit that would never occur to you but would almost certainly occur to a superintelligence, not because xe is malicious, but because they’re the optimal way to achieve the explicit goal set for xer. Your optimism, your belief that you can easily specify a goal (in computer code, not even English words) which admits of no undesirable creative shortcuts, is grossly misplaced once you bring smarter-than-human agents into the discussion. You cannot patch this problem; it has to be rigorously solved, or your AI wrecks the world.