Did anyone else immediately try to come up with ways Davis’ plan would fail? One obvious failure mode would be in specifying which dead people count—if you say “the people described in these books,” the AI could just grab the books and rewrite them. Hmm, come to think of it: is any attempt to pin down human preferences by physical reference rather than logical reference vulnerable to tampering of this kind, and therefore unworkable? I know EY has written many times before about a “giant logical function that computes morality”, but this puts that notion in a bit of a different light for me. Anyway, I’m sure there other less obvious ways Davis’ plan could go wrong too. I also suspect he’s sneaking a lot into that little word, “disapprove”.
In general though, I’m continually astounded at how many people, upon being introduced to the value loading problem and some of the pitfalls that “common-sense” approaches have, still say “Okay, but why couldn’t we just do [idea I came up with in five seconds]?”
One obvious failure mode would be in specifying which dead people count—if you say “the people described in these books,” the AI could just grab the books and rewrite them. Hmm, come to think of it: is any attempt to pin down human preferences by physical reference rather than logical reference vulnerable to tampering of this kind, and therefore unworkable?
Not as such, no. It’s a possible failure mode, similar to wireheading; but both of those are avoidable. You need to write the goal system in such a way that makes the AI care about the original referent, not any proxy that it looks at, but there’s no particular reason to think that’s impossible.
In general though, I’m continually astounded at how many people, upon being introduced to the value loading problem and some of the pitfalls that “common-sense” approaches have, still say “Okay, but why couldn’t we just do [idea I came up with in five seconds]?”
Did anyone else immediately try to come up with ways Davis’ plan would fail? One obvious failure mode would be in specifying which dead people count—if you say “the people described in these books,” the AI could just grab the books and rewrite them. Hmm, come to think of it: is any attempt to pin down human preferences by physical reference rather than logical reference vulnerable to tampering of this kind, and therefore unworkable? I know EY has written many times before about a “giant logical function that computes morality”, but this puts that notion in a bit of a different light for me. Anyway, I’m sure there other less obvious ways Davis’ plan could go wrong too. I also suspect he’s sneaking a lot into that little word, “disapprove”.
In general though, I’m continually astounded at how many people, upon being introduced to the value loading problem and some of the pitfalls that “common-sense” approaches have, still say “Okay, but why couldn’t we just do [idea I came up with in five seconds]?”
Not as such, no. It’s a possible failure mode, similar to wireheading; but both of those are avoidable. You need to write the goal system in such a way that makes the AI care about the original referent, not any proxy that it looks at, but there’s no particular reason to think that’s impossible.
Agreed.