I think I phrased my previous comment poorly. What I meant is that if you have developed a set of abstractions relevant to achieving your goals and I want to predict you accurately, then I also need to develop abstractions that are are relevant to achieving your goals. Given a limited representational capacity, this creates a pressure for you to develop representations similar to those of others.
why are the goals of other agents more likely to have natural convergent representations of them than other things in the world?
I think I phrased my previous comment poorly. What I meant is that if you have developed a set of abstractions relevant to achieving your goals and I want to predict you accurately, then I also need to develop abstractions that are are relevant to achieving your goals. Given a limited representational capacity, this creates a pressure for you to develop representations similar to those of others.