Exercise: Why does instrumental convergence happen? Would it be coherent to imagine a reality without it?
I’d say something like, there tends to be overlap between {subgoals helpful for goal X} for lots of different values of X. In the language of this sequence, there is a set of subgoals that increase the amount of attainable utility for a broad class of goals.
To imagine a reality without it, you’d need to imagine that such a set doesn’t exist. Take two different things you want, and the steps required to get there are entirely disjoint. This does seem conceivable – you can create toy universes where it’s the case – but it doesn’t describe the real world, and it’s hard to imagine that it could one day describe the real world.
I’d say something like, there tends to be overlap between {subgoals helpful for goal X} for lots of different values of X. In the language of this sequence, there is a set of subgoals that increase the amount of attainable utility for a broad class of goals.
To imagine a reality without it, you’d need to imagine that such a set doesn’t exist. Take two different things you want, and the steps required to get there are entirely disjoint. This does seem conceivable – you can create toy universes where it’s the case – but it doesn’t describe the real world, and it’s hard to imagine that it could one day describe the real world.