Carl, thanks for writing this up! I may as well unpack and say that this is pretty much how I have been thinking about the problem, too (though I hadn’t considered the idea of relative measures), and I still think I prefer biting the attendant bullets that I can see to the alternatives. But I do at least find it—well—worth pointing out that if we in fact achieve one of the higher strata, and we want to be time-consistent, it looks like we’re going to stop living our lives on the mainline probability; i.e., if the universe is of size 3^^^3, it seems like we’ll spend almost all of the available resources on trying to crack the matrix (even if there is no indication that we live in a matrix) and only an infinitesimal—combinatorially small—fraction on actually having fun.
Yes, I do think that this is probably what I will on reflection find to be the right thing, because the combinatorially small fraction pretty much looks like 3^^^3 from my current vantage point and even my middle-distance extrapolations, and as we self-modify to grow larger, since we want to be time-consistent and not regret being time-consistent, we’ll design our future selves such that we’ll keep feeling that this is the right tradeoff (i.e., this is much better than starting out with a near-certainty of not having fun at all, because our FAI puts all resources into trying to find infinite laws of physics). So perhaps it is simply appropriate (to humanity’s utility function) that immense brains spend most of their resources guarding against events of infinitesimal probabilities. But it’s sufficiently non-obvious that it at least seems worth keeping in mind.
(Also, amended the post with a note that by “4^^^^4”, I really mean “whatever is so large that it is only epsilon away from the upper bound”.)
But I do at least find it—well—worth pointing out that if we in fact achieve one of the higher strata, and we want to be time-consistent, it looks like we’re going to stop living our lives on the mainline probability;
Carl, thanks for writing this up! I may as well unpack and say that this is pretty much how I have been thinking about the problem, too (though I hadn’t considered the idea of relative measures), and I still think I prefer biting the attendant bullets that I can see to the alternatives. But I do at least find it—well—worth pointing out that if we in fact achieve one of the higher strata, and we want to be time-consistent, it looks like we’re going to stop living our lives on the mainline probability; i.e., if the universe is of size 3^^^3, it seems like we’ll spend almost all of the available resources on trying to crack the matrix (even if there is no indication that we live in a matrix) and only an infinitesimal—combinatorially small—fraction on actually having fun.
Yes, I do think that this is probably what I will on reflection find to be the right thing, because the combinatorially small fraction pretty much looks like 3^^^3 from my current vantage point and even my middle-distance extrapolations, and as we self-modify to grow larger, since we want to be time-consistent and not regret being time-consistent, we’ll design our future selves such that we’ll keep feeling that this is the right tradeoff (i.e., this is much better than starting out with a near-certainty of not having fun at all, because our FAI puts all resources into trying to find infinite laws of physics). So perhaps it is simply appropriate (to humanity’s utility function) that immense brains spend most of their resources guarding against events of infinitesimal probabilities. But it’s sufficiently non-obvious that it at least seems worth keeping in mind.
(Also, amended the post with a note that by “4^^^^4”, I really mean “whatever is so large that it is only epsilon away from the upper bound”.)
Indeed.