I don’t think I understand, what’s the reason to expect that the “acausal economy” will look like a bunch of acausal norms, as opposed to, say, each civilization first figuring out what its ultimate values are, how to encode them into a utility function, then merging with every other civilization’s utility function? (Not saying that I know it will be the latter, just that I don’t know how to tell at this point.)
Also, given that I think AI risk is very high for human civilization, and there being no reason to suspect that we’re not a typical pre-AGI civilization, most of the “acausal economy” might well consist of unaligned AIs (created accidentally by other civilizations), which makes it seemingly even harder to reason about what this “economy” looks like.
To your first question, I’m not sure which particular “the reason” would be most helpful to convey. (To contrast: what’s “the reason” that physically dispersed human societies have laws? Answer: there’s a confluence of reasons.). However, I’ll try to point out some things that might be helpful to attend to.
First, committing to a policy that merges your utility function with someone else’s is quite a vulnerable maneuver, with a lot of boundary-setting aspects. For instance, will you merge utility functions multiplicatively (as in Nash bargaining), linearly (as in Harsanyi’s utility aggregation theorem), or some other way? Also, what if the entity you’re merging with has self-modified to become a “utility monster” (an entity with strongly exaggerated preferences) so as to exploit the merging procedure? Some kind of boundary-setting is needed to decide whether, how, and how much to merge, which is one of the reasons why I think boundary-handling is more fundamental than utility-handling.
I view this as further pointing away from “just aggregate utilities” and toward “one needs to think about boundaries when aggregating beings” (see Part 1 of my Boundaries sequence). In other words, one needs (or implicitly assumes) some kind of norm about how and when to manage boundaries between utility functions, even in an abstract utility-function-merging operations where the boundary issues come down to where to draw parentheses in between additive and multiplicative operations. Thus, boundary-management are somewhat more fundamental, or conceptually upstream, of principles that might pick out a global utility function for the entirely of the “acausal society”.
(Even if the there is a global utility function that turns out to be very simple to write down, the process of verifying its agreeability will involve checking that a lot of boundary-interactions. For instance, one must check that this hypothetical reigning global utility function is not dethroned by some union of civilizations who successfully merge in opposition to it, which is a question of boundary-handling.)
What does merging utility functions look like and are you sure it’s not going to look the same as global free trade? It’s arguable that trade is just a way of breaking down and modularizing a big multifaceted problem over a lot of subagent task specialists (and there’s no avoiding having subagents, due to the light speed limit)
By the way I’d love to hear people giving my comment agreement karma explain what they’re agreeing with and how they know it’s true, because I was asking a question that I don’t know the answer to, and I really hope people don’t think that we know the answer, unless we do, in which case I’d like to hear it.
Taken literally, the only way to merge n utility functions into one without any other info (eg the preferences that generated the utility functions) is to do a weighted sum. There’s only n-1 free parameters.
So you think it’s computationally tractable? I think there are some other factors you’re missing. That’s a weighted sum of a bunch of vectors assigning numbers to all possible outcomes, either all possible histories+final states of the universe, or all possible experiences. And there are additional complications with normalizing utility functions; you don’t know the probability distribution of final outcomes (so you can’t take the integral of the utility functions) until you already know how the aggregation of normalized weighted utility functions is going to influence it.
I think the acausal economy would look aggressively space expansionist/resource-exploitative (those are the ones that will acquire and therefore control the most resources; others will self-select out or be out-competed) and, if you’re pessimistic about alignment, with some Goodharted human(-like) values from failed alignment (and possibly some bad human-like values). The Goodharting may go disproportionately in directions that are more resource-efficient and allow faster resource acquisition and use and successful takeover (against their creators and other AI). We may want to cooperate most with those using their resources disproportionately for artificial minds or for which there’s the least opportunity cost to do so (say because they’re focusing on building more hardware that could support digital minds).
I don’t think I understand, what’s the reason to expect that the “acausal economy” will look like a bunch of acausal norms, as opposed to, say, each civilization first figuring out what its ultimate values are, how to encode them into a utility function, then merging with every other civilization’s utility function? (Not saying that I know it will be the latter, just that I don’t know how to tell at this point.)
Also, given that I think AI risk is very high for human civilization, and there being no reason to suspect that we’re not a typical pre-AGI civilization, most of the “acausal economy” might well consist of unaligned AIs (created accidentally by other civilizations), which makes it seemingly even harder to reason about what this “economy” looks like.
To your first question, I’m not sure which particular “the reason” would be most helpful to convey. (To contrast: what’s “the reason” that physically dispersed human societies have laws? Answer: there’s a confluence of reasons.). However, I’ll try to point out some things that might be helpful to attend to.
First, committing to a policy that merges your utility function with someone else’s is quite a vulnerable maneuver, with a lot of boundary-setting aspects. For instance, will you merge utility functions multiplicatively (as in Nash bargaining), linearly (as in Harsanyi’s utility aggregation theorem), or some other way? Also, what if the entity you’re merging with has self-modified to become a “utility monster” (an entity with strongly exaggerated preferences) so as to exploit the merging procedure? Some kind of boundary-setting is needed to decide whether, how, and how much to merge, which is one of the reasons why I think boundary-handling is more fundamental than utility-handling.
Relatedly, Scott Garrabrant has pointed out in his sequence on geometric rationality that linear aggregation is more like not-having-a-boundary, and multiplicative aggregation is more like having-a-boundary:
https://www.lesswrong.com/posts/rc5ZKGjXTHs7wPjop/geometric-exploration-arithmetic-exploitation#The_AM_GM_Boundary
I view this as further pointing away from “just aggregate utilities” and toward “one needs to think about boundaries when aggregating beings” (see Part 1 of my Boundaries sequence). In other words, one needs (or implicitly assumes) some kind of norm about how and when to manage boundaries between utility functions, even in an abstract utility-function-merging operations where the boundary issues come down to where to draw parentheses in between additive and multiplicative operations. Thus, boundary-management are somewhat more fundamental, or conceptually upstream, of principles that might pick out a global utility function for the entirely of the “acausal society”.
(Even if the there is a global utility function that turns out to be very simple to write down, the process of verifying its agreeability will involve checking that a lot of boundary-interactions. For instance, one must check that this hypothetical reigning global utility function is not dethroned by some union of civilizations who successfully merge in opposition to it, which is a question of boundary-handling.)
What does merging utility functions look like and are you sure it’s not going to look the same as global free trade? It’s arguable that trade is just a way of breaking down and modularizing a big multifaceted problem over a lot of subagent task specialists (and there’s no avoiding having subagents, due to the light speed limit)
By the way I’d love to hear people giving my comment agreement karma explain what they’re agreeing with and how they know it’s true, because I was asking a question that I don’t know the answer to, and I really hope people don’t think that we know the answer, unless we do, in which case I’d like to hear it.
Taken literally, the only way to merge n utility functions into one without any other info (eg the preferences that generated the utility functions) is to do a weighted sum. There’s only n-1 free parameters.
So you think it’s computationally tractable? I think there are some other factors you’re missing. That’s a weighted sum of a bunch of vectors assigning numbers to all possible outcomes, either all possible histories+final states of the universe, or all possible experiences. And there are additional complications with normalizing utility functions; you don’t know the probability distribution of final outcomes (so you can’t take the integral of the utility functions) until you already know how the aggregation of normalized weighted utility functions is going to influence it.
I think the acausal economy would look aggressively space expansionist/resource-exploitative (those are the ones that will acquire and therefore control the most resources; others will self-select out or be out-competed) and, if you’re pessimistic about alignment, with some Goodharted human(-like) values from failed alignment (and possibly some bad human-like values). The Goodharting may go disproportionately in directions that are more resource-efficient and allow faster resource acquisition and use and successful takeover (against their creators and other AI). We may want to cooperate most with those using their resources disproportionately for artificial minds or for which there’s the least opportunity cost to do so (say because they’re focusing on building more hardware that could support digital minds).