We don’t really have a good theory of agent sizing and aggregation. In what cases is it “one agent with many semi-independent components” vs “many fully independent agents”? Which is more efficient, which is more morally desirable? Why not “one gigantic agent”?
In situations of competition for limited resources, of course there will be zero-sum elements. It’s not obvious that there won’t ALSO be positive-sum cooperation (to exploit non-agent areas, or to find higher sums in aggregate utility functions), until there is literally no un-optimized resources and everything is local maxima for various morally-relevant (to us) agents.
We don’t really have a good theory of agent sizing and aggregation. In what cases is it “one agent with many semi-independent components” vs “many fully independent agents”? Which is more efficient, which is more morally desirable? Why not “one gigantic agent”?
In situations of competition for limited resources, of course there will be zero-sum elements. It’s not obvious that there won’t ALSO be positive-sum cooperation (to exploit non-agent areas, or to find higher sums in aggregate utility functions), until there is literally no un-optimized resources and everything is local maxima for various morally-relevant (to us) agents.
I argue that that’s a great problem to have!