My worry is that instrumental subgoals are not safer because instrumental subgoals are automatically safer, but because higher goals (which generate instrumental subgoals) tend to generate multiple instrumental subgoals, none of which is important enough to steamroll the others. This seems to explain the cake example.
If you want instrumental goals all the way up, it means you want to repeatedly convert the highest goal into an instrumental subgoal of an even higher goal, which in turn will generate many other instrumental subgoals to compete with it for importance.
I’m not sure, but it looks like the only reason this should work is if the AGI/ASI has so many competing goals that being good to humans has some weight. This is similar to Multi-Objective Homeostasis.
Goal Reductionism
I guess another way this may work, is if the AGI/ASI itself isn’t sure why it’s doing something, we can teach it to to think that its behaviours are the instrumental subgoal of some higher purpose, which it itself can’t be sure about.
Instrumental goal competition explains the cake
My worry is that instrumental subgoals are not safer because instrumental subgoals are automatically safer, but because higher goals (which generate instrumental subgoals) tend to generate multiple instrumental subgoals, none of which is important enough to steamroll the others. This seems to explain the cake example.
If you want instrumental goals all the way up, it means you want to repeatedly convert the highest goal into an instrumental subgoal of an even higher goal, which in turn will generate many other instrumental subgoals to compete with it for importance.
I’m not sure, but it looks like the only reason this should work is if the AGI/ASI has so many competing goals that being good to humans has some weight. This is similar to Multi-Objective Homeostasis.
Goal Reductionism
I guess another way this may work, is if the AGI/ASI itself isn’t sure why it’s doing something, we can teach it to to think that its behaviours are the instrumental subgoal of some higher purpose, which it itself can’t be sure about.
This is related to Goal Reductionism.
I feel that Self-Other Overlap: A Neglected Approach to AI Alignment also fits the theme of the chef and restaurant example, and may help with Goal Reductionism.