To your first question, I’m not sure which particular “the reason” would be most helpful to convey. (To contrast: what’s “the reason” that physically dispersed human societies have laws? Answer: there’s a confluence of reasons.). However, I’ll try to point out some things that might be helpful to attend to.
First, committing to a policy that merges your utility function with someone else’s is quite a vulnerable maneuver, with a lot of boundary-setting aspects. For instance, will you merge utility functions multiplicatively (as in Nash bargaining), linearly (as in Harsanyi’s utility aggregation theorem), or some other way? Also, what if the entity you’re merging with has self-modified to become a “utility monster” (an entity with strongly exaggerated preferences) so as to exploit the merging procedure? Some kind of boundary-setting is needed to decide whether, how, and how much to merge, which is one of the reasons why I think boundary-handling is more fundamental than utility-handling.
I view this as further pointing away from “just aggregate utilities” and toward “one needs to think about boundaries when aggregating beings” (see Part 1 of my Boundaries sequence). In other words, one needs (or implicitly assumes) some kind of norm about how and when to manage boundaries between utility functions, even in an abstract utility-function-merging operations where the boundary issues come down to where to draw parentheses in between additive and multiplicative operations. Thus, boundary-management are somewhat more fundamental, or conceptually upstream, of principles that might pick out a global utility function for the entirely of the “acausal society”.
(Even if the there is a global utility function that turns out to be very simple to write down, the process of verifying its agreeability will involve checking that a lot of boundary-interactions. For instance, one must check that this hypothetical reigning global utility function is not dethroned by some union of civilizations who successfully merge in opposition to it, which is a question of boundary-handling.)
To your first question, I’m not sure which particular “the reason” would be most helpful to convey. (To contrast: what’s “the reason” that physically dispersed human societies have laws? Answer: there’s a confluence of reasons.). However, I’ll try to point out some things that might be helpful to attend to.
First, committing to a policy that merges your utility function with someone else’s is quite a vulnerable maneuver, with a lot of boundary-setting aspects. For instance, will you merge utility functions multiplicatively (as in Nash bargaining), linearly (as in Harsanyi’s utility aggregation theorem), or some other way? Also, what if the entity you’re merging with has self-modified to become a “utility monster” (an entity with strongly exaggerated preferences) so as to exploit the merging procedure? Some kind of boundary-setting is needed to decide whether, how, and how much to merge, which is one of the reasons why I think boundary-handling is more fundamental than utility-handling.
Relatedly, Scott Garrabrant has pointed out in his sequence on geometric rationality that linear aggregation is more like not-having-a-boundary, and multiplicative aggregation is more like having-a-boundary:
https://www.lesswrong.com/posts/rc5ZKGjXTHs7wPjop/geometric-exploration-arithmetic-exploitation#The_AM_GM_Boundary
I view this as further pointing away from “just aggregate utilities” and toward “one needs to think about boundaries when aggregating beings” (see Part 1 of my Boundaries sequence). In other words, one needs (or implicitly assumes) some kind of norm about how and when to manage boundaries between utility functions, even in an abstract utility-function-merging operations where the boundary issues come down to where to draw parentheses in between additive and multiplicative operations. Thus, boundary-management are somewhat more fundamental, or conceptually upstream, of principles that might pick out a global utility function for the entirely of the “acausal society”.
(Even if the there is a global utility function that turns out to be very simple to write down, the process of verifying its agreeability will involve checking that a lot of boundary-interactions. For instance, one must check that this hypothetical reigning global utility function is not dethroned by some union of civilizations who successfully merge in opposition to it, which is a question of boundary-handling.)