I expect it to act a lot like probability distributions and utility functions in the “local” situation after the reasoner has chunked the world.
I agree with this: (i) it feels true and would be surprising not to add up to normality, (ii) coherence theorems suggest that any preferences can be represented as probabilities+utilities in the case of finitely many outcomes.
“utility is a number, not a function”
This is my view as well, but you still need to handle the dependence on subjective uncertainty. I think the core thing at issue is whether that uncertainty is represented by a probability distribution (where utility is an expectation).
(Slightly less important: my most naive guess is that the utility number is itself represented as a sum over objects, and then we might use “utility function” to refer to the thing being summed.)
Also, to say a bit more on why I’m not sold that the current situation is divergent in the St. Petersburg way wrt, eg, amount of Fun...
I don’t mean that we face some small chance of encountering a St Petersburg lottery. I mean that when I actually think about the scale of the universe, and what I ought to believe about physics, I just immediately run into St Petersburg-style cases:
It’s unclear whether we can have an extraordinarily long-lived civilization if we reduce entropy consumption to ~0 (e.g. by having a reversible civilization). That looks like at least 5% probability, and would suggest the number of happy lives is much more than 10100 times larger than I might have thought. So does it dominate the expectation?
But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe. Maybe that happens with only 1% probability, but it corresponds to yet bigger civilization. So does that mean we should think that colonizing faster increases the value of the future by 1%, or by 100% since these possibilities are bigger and better and dominate the expectation?
But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it (but a large fraction of them are nevertheless filled with other civilizations like ours). Perhaps that’s 20%. So it actually looks more likely than the “long-lived reversible civilization” and implies more total flourishing. And on those perspectives not going extinct is more important than going faster, for the usual reasons. So does that dominate the calculus instead?
Perhaps rather than having a single set of physical constants, our universe runs every possible set. If that’s 5%, and could stack on top of any of the above while implying another factor of 10100 of scale. And if the standard model had no magic constants maybe this possibility would be 1% instead of 5%. So should I updated by a factor of 5 that we won’t discover that the standard model has fewer magic constants, because then “continuum of possible copies of our universe running in parallel” has only 1% chance instead of 5%?
Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant 10100 more people, then it starts to seem like even less of a further ask to assume that there were actually 101000 more people instead. So should we assume that multiple of those enhugening assumptions are true (since each one increases values by more than it decreases probability), or just take our favorite and then keep cranking up the numbers larger and larger (with each cranking being more probable than the last and hence more probable than adding a second enhugening assumption)?
Those are very naive physically statements of the possibilities, but the point is that it seems easy to imagine the possibility that populations could be vastly larger than we think “by default”, and many of those possibilities seem to have reasonable chances rather than being vanishingly unlikely. And at face value you might have thought those possibilities were actually action-relevant (e.g. the possibility of exponential returns to resources dominates the EV and means we should rush to colonize after all), but once you actually look at the whole menu, and see how the situation is just obviously paradoxical in every dimension, I think it’s pretty clear that you should cut off this line of thinking.
A bit more precisely: this situation is structurally identical to someone in a St Petersburg paradox shuffling around the outcomes and finding that they can justify arbitrary comparisons because everything has infinite EV and it’s easy to rewrite. That is, we can match each universe U with “U but with long-lived reversible civilizations,” and we find that the long-lived reversible civilizations dominate the calculus. Or we can match each universe U with “U but vast” and find the vast universes dominate the calculus. Or we can match “long-lived reversible civilizations” with “vast” and find that we can ignore long-lived reversible civilizations. It’s just like matching up the outcomes in the St Petersburg paradox in order to show that any outcome dominates itself.
The unbounded utility claim seems precisely like the claim that each of those less-likely-but-larger universes ought to dominate our concern, compared to the smaller-but-more-likely universe we expect by default. And that way of reasoning seems like it leads directly to these contradictions at the very first time you try to apply it to our situation (indeed, I think every time I’ve seen someone use this assumption in a substantive way it has immediately seemed to run into paradoxes, which are so severe that they mean they could as well have reached the opposite conclusion by superficially-equally-valid reasoning).
I totally believe you might end up with a very different way of handling big universes than “bounded utilities,” but I suspect it will also lead to the conclusion that “the plausible prospect of a big universe shouldn’t dominate our concern.” And I’d probably be fine with the result. Once you divorce unbounded utilities from the usual theory about how utilities work, and also divorce them from what currently seems like their main/only implication, I expect I won’t have anything more than a semantic objection.
More specifically and less confidently, I do think there’s a pretty good chance that whatever theory you end up with will agree roughly with the way that I handle big universes—we’ll just use our real probabilities of each of these universes rather than focusing on the big ones in virtue of their bigness, and within each universe we’ll still prefer have larger flourishing populations. I do think that conclusion is fairly uncertain, but I tentatively think it’s more likely we’ll give up on the principle “a bigger civilization is nearly-linearly better within a given universe” than on the principle “a bigger universe is much less than linearly more important.”
And from a practical perspective, I’m not sure what interim theory you use to reason about these things. I suspect it’s mostly academic for you because e.g. you think alignment is a 99% risk of death instead of a 20% risk of death and hence very few other questions about the future matter. But if you ever did find yourself having to reason about humanity’s long-term future (e.g. to assess the value of extinction risk vs faster colonization, or the extent of moral convergence), then it seems like you should use an interim theory which isn’t fanatical about the possibility of big universes—because the fanatical theories just don’t work, and spit out inconsistent results if combined with our current framework. You can also interpret my argument as strongly objecting to the use of unbounded utilities in that interim framework.
(I, in fact, lifted it off of you, a number of years ago :-p)
but you still need to handle the dependence on subjective uncertainty.
Of course. (And noting that I am, perhaps, more openly confused about how to handle the subjective uncertainty than you are, given my confusions around things like logical uncertainty and whether difficult-to-normalize arithmetical expressions meaningfully denote numbers.)
Running through your examples:
It’s unclear whether we can have an extraordinarily long-lived civilization …
I agree. Separately, I note that I doubt total Fun is linear in how much compute is available to civilization; continuity with the past & satisfactory completion of narrative arcs started in the past is worth something, from which we deduce that wiping out civilization and replacing it with another different civilization of similar flourish and with 2x as much space to flourish in, is not 2x as good as leaving the original civilization alone. But I’m basically like “yep, whether we can get reversibly-computed Fun chugging away through the high-entropy phase of the universe seems like an empiricle question with cosmically large swings in utility associated therewith.”
But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe.
This seems fairly plausible to me! For instance, my best guess is that you can get more than 2x the Fun by computing two people interacting than by computing two individuals separately. (Although my best guess is also that this effect diminishes at scale, \shrug.)
By my lights, it sure would be nice to have more clarity on this stuff before needing to decide how much to rush our expansion. (Although, like, 1st world problems.)
But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it
Sure, this is pretty plausible, but (arguendo) it shouldn’t really be factoring into our action analysis, b/c of the part where we can’t reach it. \shrug
Perhaps rather than having a single set of physical constants, our universe runs every possible set.
Sure. And again (arguendo) this doesn’t much matter to us b/c the others are beyond our sphere of influence.
Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant 10^100 more people, then it starts to seem like even less of a further ask to assume that there were actually 10^1000 more people instead
I think this is where I get off the train (at least insofar as I entertain unbounded-utility hypotheses). Like, our ability to reversibly compute in the high-entropy regime is bounded by our error-correction capabilities, and we really start needing to upend modern physics as I understand it to make the numbers really huge. (Like, maybe 10^1000 is fine, but it’s gonna fall off a cliff at some point.)
I have a sense that I’m missing some deeper point you’re trying to make.
I also have a sense that… how to say… like, suppose someone argued “well, you don’t have 1/∞ probability that “infinite utility” makes sense, so clearly you’ve got to take infinite utilities seriously”. My response would be something like “That seems mixed up to me. Like, on my current understanding, “infinite utility” is meaningless, it’s a confusion, and I just operate day-to-day without worrying about it. It’s not so much that my operating model assigns probability 0 to the proposition “infinite utilities are meaningful”, as that infinite utilities simply don’t fit into my operating model, they don’t make sense, they don’t typecheck. And separately, I’m not yet philosophically mature, and I can give you various meta-probabilities about what sorts of things will and won’t typecheck in my operating model tomorrow. And sure, I’m not 100% certain that we’ll never find a way to rescue the idea of infinite utilities. But that meta-uncertainty doesn’t bleed over into my operating model, and I’m not supposed to ram infinities into a place where they don’t fit just b/c I might modify the type signatures tomorrow.”
When you bandy around plausible ways that the universe could be real large, it doesn’t look obviously divergent to me. Some of the bullets you’re handling are ones that I am just happy to bite, and others involve stuff that I’m not sure I’m even going to think will typecheck, once I understand wtf is going on. Like, just as I’m not compelled by “but you have more than 0% probability that ‘infinite utility’ is meaningful” (b/c it’s mixing up the operating model and my philosophical immaturity), I’m not compelled by “but your operating model, which says that X, Y, and Z all typecheck, is badly divergent”. Yeah, sure, and maybe the resolution is that utilities are bounded, or maybe it’s that my operating model is too permissive on account of my philosophical immaturity. Philosophical immaturity can lead to an operating model that’s too permisive (cf. zombie arguments) just as easily as one that’s too strict.
Like… the nature of physical law keeps seeming to play games like “You have continua!! But you can’t do an arithmetic encoding. There’s infinite space!! But most of it is unreachable. Time goes on forever!! But most of it is high-entropy. You can do reversible computing to have Fun in a high-entropy universe!! But error accumulates.” And this could totally be a hint about how things that are real can’t help but avoid the truly large numbers (never mind the infinities), or something, I don’t know, I’m philisophically immature. But from my state of philosophical immaturity, it looks like this could totally still resolve in a “you were thinking about it wrong; the worst enhugening assumptions fail somehow to typecheck” sort of way.
Trying to figure out the point that you’re making that I’m missing, it sounds like you’re trying to say something like “Everyday reasoning at merely-cosmic scales already diverges, even without too much weird stuff. We already need to bound our utilities, when we shift from looking at the milk in the supermarket to looking at the stars in the sky (nevermind the rest of the mathematical multiverse, if there is such a thing).” Is that about right?
If so, I indeed do not yet buy it. Perhaps spell it out in more detail, for someone who’s suspicious of any appeals to large swaths of terrain that we can’t affect (eg, variants of this universe w/ sufficiently different cosmological constants, at least in the regions where the locals aren’t thinking about us-in-particular); someone who buys reversible computing but is going to get suspicious when you try to drive the error rate to shockingly low lows?
To be clear, insofar as modern cosmic-scale reasoning diverges (without bringing in considerations that I consider suspicious and that I suspect I might later think belong in the ‘probably not meaningful (in the relevant way)’ bin), I do start to feel the vice grips on me, and I expect I’d give bounded utilities another look if I got there.
I agree with this: (i) it feels true and would be surprising not to add up to normality, (ii) coherence theorems suggest that any preferences can be represented as probabilities+utilities in the case of finitely many outcomes.
This is my view as well, but you still need to handle the dependence on subjective uncertainty. I think the core thing at issue is whether that uncertainty is represented by a probability distribution (where utility is an expectation).
(Slightly less important: my most naive guess is that the utility number is itself represented as a sum over objects, and then we might use “utility function” to refer to the thing being summed.)
I don’t mean that we face some small chance of encountering a St Petersburg lottery. I mean that when I actually think about the scale of the universe, and what I ought to believe about physics, I just immediately run into St Petersburg-style cases:
It’s unclear whether we can have an extraordinarily long-lived civilization if we reduce entropy consumption to ~0 (e.g. by having a reversible civilization). That looks like at least 5% probability, and would suggest the number of happy lives is much more than 10100 times larger than I might have thought. So does it dominate the expectation?
But nearly-reversible civilizations can also have exponential returns to the resources they are able to acquire during the messy phase of the universe. Maybe that happens with only 1% probability, but it corresponds to yet bigger civilization. So does that mean we should think that colonizing faster increases the value of the future by 1%, or by 100% since these possibilities are bigger and better and dominate the expectation?
But also it seems quite plausible that our universe is already even-more-exponentially spatially vast, and we merely can’t reach parts of it (but a large fraction of them are nevertheless filled with other civilizations like ours). Perhaps that’s 20%. So it actually looks more likely than the “long-lived reversible civilization” and implies more total flourishing. And on those perspectives not going extinct is more important than going faster, for the usual reasons. So does that dominate the calculus instead?
Perhaps rather than having a single set of physical constants, our universe runs every possible set. If that’s 5%, and could stack on top of any of the above while implying another factor of 10100 of scale. And if the standard model had no magic constants maybe this possibility would be 1% instead of 5%. So should I updated by a factor of 5 that we won’t discover that the standard model has fewer magic constants, because then “continuum of possible copies of our universe running in parallel” has only 1% chance instead of 5%?
Why not all of the above? What if the universe is vast and it allows for very long lived civilization? And once we bite any of those bullets to grant 10100 more people, then it starts to seem like even less of a further ask to assume that there were actually 101000 more people instead. So should we assume that multiple of those enhugening assumptions are true (since each one increases values by more than it decreases probability), or just take our favorite and then keep cranking up the numbers larger and larger (with each cranking being more probable than the last and hence more probable than adding a second enhugening assumption)?
Those are very naive physically statements of the possibilities, but the point is that it seems easy to imagine the possibility that populations could be vastly larger than we think “by default”, and many of those possibilities seem to have reasonable chances rather than being vanishingly unlikely. And at face value you might have thought those possibilities were actually action-relevant (e.g. the possibility of exponential returns to resources dominates the EV and means we should rush to colonize after all), but once you actually look at the whole menu, and see how the situation is just obviously paradoxical in every dimension, I think it’s pretty clear that you should cut off this line of thinking.
A bit more precisely: this situation is structurally identical to someone in a St Petersburg paradox shuffling around the outcomes and finding that they can justify arbitrary comparisons because everything has infinite EV and it’s easy to rewrite. That is, we can match each universe U with “U but with long-lived reversible civilizations,” and we find that the long-lived reversible civilizations dominate the calculus. Or we can match each universe U with “U but vast” and find the vast universes dominate the calculus. Or we can match “long-lived reversible civilizations” with “vast” and find that we can ignore long-lived reversible civilizations. It’s just like matching up the outcomes in the St Petersburg paradox in order to show that any outcome dominates itself.
The unbounded utility claim seems precisely like the claim that each of those less-likely-but-larger universes ought to dominate our concern, compared to the smaller-but-more-likely universe we expect by default. And that way of reasoning seems like it leads directly to these contradictions at the very first time you try to apply it to our situation (indeed, I think every time I’ve seen someone use this assumption in a substantive way it has immediately seemed to run into paradoxes, which are so severe that they mean they could as well have reached the opposite conclusion by superficially-equally-valid reasoning).
I totally believe you might end up with a very different way of handling big universes than “bounded utilities,” but I suspect it will also lead to the conclusion that “the plausible prospect of a big universe shouldn’t dominate our concern.” And I’d probably be fine with the result. Once you divorce unbounded utilities from the usual theory about how utilities work, and also divorce them from what currently seems like their main/only implication, I expect I won’t have anything more than a semantic objection.
More specifically and less confidently, I do think there’s a pretty good chance that whatever theory you end up with will agree roughly with the way that I handle big universes—we’ll just use our real probabilities of each of these universes rather than focusing on the big ones in virtue of their bigness, and within each universe we’ll still prefer have larger flourishing populations. I do think that conclusion is fairly uncertain, but I tentatively think it’s more likely we’ll give up on the principle “a bigger civilization is nearly-linearly better within a given universe” than on the principle “a bigger universe is much less than linearly more important.”
And from a practical perspective, I’m not sure what interim theory you use to reason about these things. I suspect it’s mostly academic for you because e.g. you think alignment is a 99% risk of death instead of a 20% risk of death and hence very few other questions about the future matter. But if you ever did find yourself having to reason about humanity’s long-term future (e.g. to assess the value of extinction risk vs faster colonization, or the extent of moral convergence), then it seems like you should use an interim theory which isn’t fanatical about the possibility of big universes—because the fanatical theories just don’t work, and spit out inconsistent results if combined with our current framework. You can also interpret my argument as strongly objecting to the use of unbounded utilities in that interim framework.
(I, in fact, lifted it off of you, a number of years ago :-p)
Of course. (And noting that I am, perhaps, more openly confused about how to handle the subjective uncertainty than you are, given my confusions around things like logical uncertainty and whether difficult-to-normalize arithmetical expressions meaningfully denote numbers.)
Running through your examples:
I agree. Separately, I note that I doubt total Fun is linear in how much compute is available to civilization; continuity with the past & satisfactory completion of narrative arcs started in the past is worth something, from which we deduce that wiping out civilization and replacing it with another different civilization of similar flourish and with 2x as much space to flourish in, is not 2x as good as leaving the original civilization alone. But I’m basically like “yep, whether we can get reversibly-computed Fun chugging away through the high-entropy phase of the universe seems like an empiricle question with cosmically large swings in utility associated therewith.”
This seems fairly plausible to me! For instance, my best guess is that you can get more than 2x the Fun by computing two people interacting than by computing two individuals separately. (Although my best guess is also that this effect diminishes at scale, \shrug.)
By my lights, it sure would be nice to have more clarity on this stuff before needing to decide how much to rush our expansion. (Although, like, 1st world problems.)
Sure, this is pretty plausible, but (arguendo) it shouldn’t really be factoring into our action analysis, b/c of the part where we can’t reach it. \shrug
Sure. And again (arguendo) this doesn’t much matter to us b/c the others are beyond our sphere of influence.
I think this is where I get off the train (at least insofar as I entertain unbounded-utility hypotheses). Like, our ability to reversibly compute in the high-entropy regime is bounded by our error-correction capabilities, and we really start needing to upend modern physics as I understand it to make the numbers really huge. (Like, maybe 10^1000 is fine, but it’s gonna fall off a cliff at some point.)
I have a sense that I’m missing some deeper point you’re trying to make.
I also have a sense that… how to say… like, suppose someone argued “well, you don’t have 1/∞ probability that “infinite utility” makes sense, so clearly you’ve got to take infinite utilities seriously”. My response would be something like “That seems mixed up to me. Like, on my current understanding, “infinite utility” is meaningless, it’s a confusion, and I just operate day-to-day without worrying about it. It’s not so much that my operating model assigns probability 0 to the proposition “infinite utilities are meaningful”, as that infinite utilities simply don’t fit into my operating model, they don’t make sense, they don’t typecheck. And separately, I’m not yet philosophically mature, and I can give you various meta-probabilities about what sorts of things will and won’t typecheck in my operating model tomorrow. And sure, I’m not 100% certain that we’ll never find a way to rescue the idea of infinite utilities. But that meta-uncertainty doesn’t bleed over into my operating model, and I’m not supposed to ram infinities into a place where they don’t fit just b/c I might modify the type signatures tomorrow.”
When you bandy around plausible ways that the universe could be real large, it doesn’t look obviously divergent to me. Some of the bullets you’re handling are ones that I am just happy to bite, and others involve stuff that I’m not sure I’m even going to think will typecheck, once I understand wtf is going on. Like, just as I’m not compelled by “but you have more than 0% probability that ‘infinite utility’ is meaningful” (b/c it’s mixing up the operating model and my philosophical immaturity), I’m not compelled by “but your operating model, which says that X, Y, and Z all typecheck, is badly divergent”. Yeah, sure, and maybe the resolution is that utilities are bounded, or maybe it’s that my operating model is too permissive on account of my philosophical immaturity. Philosophical immaturity can lead to an operating model that’s too permisive (cf. zombie arguments) just as easily as one that’s too strict.
Like… the nature of physical law keeps seeming to play games like “You have continua!! But you can’t do an arithmetic encoding. There’s infinite space!! But most of it is unreachable. Time goes on forever!! But most of it is high-entropy. You can do reversible computing to have Fun in a high-entropy universe!! But error accumulates.” And this could totally be a hint about how things that are real can’t help but avoid the truly large numbers (never mind the infinities), or something, I don’t know, I’m philisophically immature. But from my state of philosophical immaturity, it looks like this could totally still resolve in a “you were thinking about it wrong; the worst enhugening assumptions fail somehow to typecheck” sort of way.
Trying to figure out the point that you’re making that I’m missing, it sounds like you’re trying to say something like “Everyday reasoning at merely-cosmic scales already diverges, even without too much weird stuff. We already need to bound our utilities, when we shift from looking at the milk in the supermarket to looking at the stars in the sky (nevermind the rest of the mathematical multiverse, if there is such a thing).” Is that about right?
If so, I indeed do not yet buy it. Perhaps spell it out in more detail, for someone who’s suspicious of any appeals to large swaths of terrain that we can’t affect (eg, variants of this universe w/ sufficiently different cosmological constants, at least in the regions where the locals aren’t thinking about us-in-particular); someone who buys reversible computing but is going to get suspicious when you try to drive the error rate to shockingly low lows?
To be clear, insofar as modern cosmic-scale reasoning diverges (without bringing in considerations that I consider suspicious and that I suspect I might later think belong in the ‘probably not meaningful (in the relevant way)’ bin), I do start to feel the vice grips on me, and I expect I’d give bounded utilities another look if I got there.