This is correct, and it is a reason why orthogonality is not likely to be true in practice, in the sense that it will probably not be easy to make an intelligent computer pursue just any random goal.
Basically, an architecture which allows you to plug any random goal into it, is like a situation where you make someone a slave and demand that he act for the sake of your goal. The slave is intelligent and already has goals, so there is no way you can guarantee that he will do what you want. Instead, he may escape and pursue his own goals. In the same way, an architecture that allows for plugging in random goals implies that the intelligence is already there, with other goals, and you are simply compelling it to pursue the goal you want. But since it is already intelligent it may escape and pursue its own goals.
This is correct, and it is a reason why orthogonality is not likely to be true in practice, in the sense that it will probably not be easy to make an intelligent computer pursue just any random goal.
Basically, an architecture which allows you to plug any random goal into it, is like a situation where you make someone a slave and demand that he act for the sake of your goal. The slave is intelligent and already has goals, so there is no way you can guarantee that he will do what you want. Instead, he may escape and pursue his own goals. In the same way, an architecture that allows for plugging in random goals implies that the intelligence is already there, with other goals, and you are simply compelling it to pursue the goal you want. But since it is already intelligent it may escape and pursue its own goals.