It’s like the difference between a tricky djinn and the ‘ends in gry’ guy.
Right, but the point is, a real-life UFAI isn’t going to have a utility function derived from a human’s verbal command. If it did, you could just order the genie to implement CEV, or shout “I call for my values to be fulfilled!”, and it would work. That’s thinking of AI in terms of sorcery rather than science.
According to my personal knowledge, various means of building AI preference functions might be employed, since research has found that the learning algorithms necessary to acquire knowledge and understanding are quite separate from decision-making algorithms necessary to start paper-clipping. Building an AI might actually consist of “train the learner for a year on corpora from human culture, develop an induced ‘internal programming language’, and only afterwards add a decision-making algorithm with a utility function phrased in terms of the induced concepts, which may as well include ‘goodness’”.
Right, but the point is, a real-life UFAI isn’t going to have a utility function derived from a human’s verbal command. If it did, you could just order the genie to implement CEV, or shout “I call for my values to be fulfilled!”, and it would work. That’s thinking of AI in terms of sorcery rather than science.
According to my personal knowledge, various means of building AI preference functions might be employed, since research has found that the learning algorithms necessary to acquire knowledge and understanding are quite separate from decision-making algorithms necessary to start paper-clipping. Building an AI might actually consist of “train the learner for a year on corpora from human culture, develop an induced ‘internal programming language’, and only afterwards add a decision-making algorithm with a utility function phrased in terms of the induced concepts, which may as well include ‘goodness’”.
This carries its own problems.
I hope you noticed that your objection and mine are pointing in the same direction.