Using the perspective from The ground of optimization means you can get rid of the action space and just say “given some prior and some utility function, what percentile of the distribution does this system tend to evolve towards?” (where the optimization power is again the log of this percentile)
We might then say that an optimizing system is compatible with strategy stealing if it’s retargetable for a wide set of utility functions in a way that produces an optimizing system that has the same amount of optimization power.
An AI that is compatible with strategy stealing is one such way to of producing an optimizing system that is compatible with strategy stealing with a particularly easy form of retargeting, but the difficulty of retargeting provides another useful dimension along which optimizing systems vary, e.g. instead of the optimization power the AI can direct towards different goals, you have a space of the “size” of allowed retargeting and the optimization power applied toward the goal for all goals and retargeting sizes.
Using the perspective from The ground of optimization means you can get rid of the action space and just say “given some prior and some utility function, what percentile of the distribution does this system tend to evolve towards?” (where the optimization power is again the log of this percentile)
We might then say that an optimizing system is compatible with strategy stealing if it’s retargetable for a wide set of utility functions in a way that produces an optimizing system that has the same amount of optimization power.
An AI that is compatible with strategy stealing is one such way to of producing an optimizing system that is compatible with strategy stealing with a particularly easy form of retargeting, but the difficulty of retargeting provides another useful dimension along which optimizing systems vary, e.g. instead of the optimization power the AI can direct towards different goals, you have a space of the “size” of allowed retargeting and the optimization power applied toward the goal for all goals and retargeting sizes.