Kind of an aside, but I think it’s underappreciated (including in economics and game theory) how much humans actually can do this sort of ‘exotic’ thing. We totally merge utility functions a bit all the time! With friends and family and colleagues and acquaintances. Crudely modelling, we have something like an affection/altruism coefficient for people we recognise (and even for members of abstract groups/coalitions we recognise or conceptualise). And besides this innate thing, we formally and normatively erect firms, institutions etc which embody heuristically merged preference mappings and so on etc.
I’m not aware of useful theory which relates to this.
We’ll go from individual minds to super-minds, just-out minds.
So, I don’t think this really a qualitative change.
In sad cases, humans as constituents of super-minds go extinct, and machine minds (and coalitions of minds, and overlapping coalitions of coalitions etc.) continue to exhibit super-minding. In happy cases, humans as willing and participant members of super-minds continue to flourish, in part by virtue of the super-minds’ competence and coherence, and in part on account of a sufficient internal balance of liberality and temperance.
An aside: I think sometimes conceptualisation of super-coordination or super-minds etc. is unfortunately quite hierarchical[1], quite feudalistic. I tentatively think modern humans benefit a lot from belonging to overlapping coalitions and communities, unlike the analogy to multi-gene genomes or multi-organelle cells or multi-cellular organisms. And in any case, it looks pretty difficult and harmful to go from where we are today to a more rigidly tree-like structure of social relations, even if humans could live just fine or even flourish in such conditions.
I don’t mean this in a sort of pejorative ‘power relations/inequality’ way, I mean in the ‘structured like a tree’ way, where there aren’t overlaps or cross-links between subcommunities.
Kind of an aside, but I think it’s underappreciated (including in economics and game theory) how much humans actually can do this sort of ‘exotic’ thing. We totally merge utility functions a bit all the time! With friends and family and colleagues and acquaintances. Crudely modelling, we have something like an affection/altruism coefficient for people we recognise (and even for members of abstract groups/coalitions we recognise or conceptualise). And besides this innate thing, we formally and normatively erect firms, institutions etc which embody heuristically merged preference mappings and so on etc.
I’m not aware of useful theory which relates to this.
So, I don’t think this really a qualitative change.
In sad cases, humans as constituents of super-minds go extinct, and machine minds (and coalitions of minds, and overlapping coalitions of coalitions etc.) continue to exhibit super-minding. In happy cases, humans as willing and participant members of super-minds continue to flourish, in part by virtue of the super-minds’ competence and coherence, and in part on account of a sufficient internal balance of liberality and temperance.
An aside: I think sometimes conceptualisation of super-coordination or super-minds etc. is unfortunately quite hierarchical[1], quite feudalistic. I tentatively think modern humans benefit a lot from belonging to overlapping coalitions and communities, unlike the analogy to multi-gene genomes or multi-organelle cells or multi-cellular organisms. And in any case, it looks pretty difficult and harmful to go from where we are today to a more rigidly tree-like structure of social relations, even if humans could live just fine or even flourish in such conditions.
I don’t mean this in a sort of pejorative ‘power relations/inequality’ way, I mean in the ‘structured like a tree’ way, where there aren’t overlaps or cross-links between subcommunities.