Ah I just left basically the same pointer haha.
Though I think the Boddy-Carroll-Pollack argument is separate from the swampland argument that Mitchell Porter is referencing. The swampland argument is saying that long-lived de Sitter space might not be possible at all in quantum gravity, whereas the Boddy-Carroll-Pollack argument is saying that long-lived de Sitter space might not give rise to Boltzmann Brains at all.
There is also some debate over whether eternal de Sitter space would even exhibit the right kind of fluctuations to support Boltzmann Brains at all. See Boddy, Carroll & Pollack (2014) for a standard argument.
To be clear, it’s not “Yudkowsky’s paradox” as Yudkowsky is explaining a standard and known problem in cosmology here.
Admittedly, many cosmologists are not super careful when reasoning about Boltzmann Brains and they also disagree about what exactly is the problem they pose (e.g., see this dialectic between Carroll (2017) and Elga (2025) about whether belief in Boltzmann Brains leads to “cognitive instability”). But the core phenomenon of Boltzmann Brains and that they seem like bad news for many cosmological models is well-understood.
Indexical uncertainty is irreducible subjective uncertainty induced by anthropically expecting to be in more than one possible future world.
I think the phrase “more than one possible future world” is misleading and should be changed. The paradigmatic cases of indexical uncertainty (including both given examples of the five-AI-clones case and of Everettian quantum mechanics) involve no certainty about which objective possible world is actual, and only involves self-locating uncertainty about my location within a possible world.
So it would be more accurate to say that it is irreducible subjective uncertainty over multiple centered possible worlds or multiple locations with a given possible world.
I agree that the point you made in the post is true and important.
I think the reason that people find alleged coherence arguments compelling is more directly that they agree with the axioms of the theorems when applied to real-world cases. For very many tasks that people want performed in real life, the success criterion is in fact complete and transitive over outcomes (e.g., more profit is better), or just directly defined via an expected utility over outcomes (e.g., Coefficient Giving explicitly aims to make grants with the highest expected impact per dollar).
I think it would be helpful if you could point to some illustrative examples of what it would concretely look like for e.g., POST-agents to perform well in such real-world tasks while remaining shutdownable.
For me at least, this picture just doesn’t capture the phenomenon at hand.
The data I have are certain first-personal facts of the form “there is a stream of experience playing before me, and the contents of this stream exhibit certain features and regularities”, and I am trying to explain why I am experiencing this particular inner movie in my head rather than some other inner movie. (If you want to dispense with talk of continuity over time and subjective anticipation, then replace all talk of “stream of experience”/”inner movie” with “snapshot of experience”/”inner photo”.)
“You in fact subjectively care more about certain experiences” does not seem like the right kind of answer to the question of “why these/this experience?”, in the same way that “you in fact subjectively care more about ordered universes” does not seem like the right kind of answer to the problem of induction.
Nice Dutch book! Vince Conitzer actually gives an almost identical Dutch book in Section 5 of his article on Dutch books against EDT.
His setup is the following: there are two Sleeping Beauties. Two independent fair coins determine the creation of rooms. The first fair coin determines whether a white or black room is created, and if a second fair coin lands heads, a room of the opposite color to the first room is created. So 1⁄4 chance there is one white room, 1⁄4 chance there is one black room, and 1⁄2 chance there is both a white room and a black room. On Monday, one Beauty is always awoken in a white room and the other Beauty is always awoken in a black room, and if their designated room was not created, they stay asleep. They bet under a joint account whose value they are both trying to maximize.
His Dutch book consists of the following bets:
Translating to your example: “wake up in a white room” = “prove Y1“, “wake up in a black room” = “prove Y2”, W = “only Y1 is true”, B = “only Y2 is true”, and WB = “both Y1 and Y2 are true, i.e., X is true”.
The only difference is that the probabilities are Pr(W)=1/4, Pr(B)=1/4, and Pr(WB)=1/2 for Conitzer and Pr(W)=1/4, Pr(B)=1/4, and Pr(WB)=1/4 in your case. So the Dutch book ends up being the same, once you appropriately adjust the stakes for the different probability placed on the “WB” scenario.
The key is once again that the “interpretation of evidential decision theory comes down to requiring that each Beauty, when calculating her expected utility from accepting a bet (or declining it), assumes in this calculation that the other Beauty would do the same”. The justification give by Conitzer is that their epistemic situations (including bets faced) are entirely symmetric.
As described here, EDT agents will act in very strange ways if they also update their prior upon observing evidence.
I’ve seen this post cited a few times now in support of the idea that “EDT double counts”. But the betting behaviour in Calculator Bet seems just straightforwardly and intuitively correct to me (from an EDT perspective).
Here’s what seems like an analogous case: 0.5 prior on X being true. I look at a calculator that is 0.99 reliable. After I make my observation, if my observation is veridical (e.g., seeing “X is true” when X is actually true) then with 0.99 chance someone clones me N times, and with 0.01 chance nothing happens; if my observation is deceptive (e.g., seeing “X is true” when X is actually false) then with 0.01 chance someone clones me N times, and with 0.99 chance nothing happens.
In this case, like in the original Calculator Bet, I should “bet at 9999:1 odds”, but in both cases that’s because seeing “X is true” has two real updates: (i) X is probably true, and (ii) there are probably many clones of me. I feel like it’s because the odds are 0.99 in both parts that makes this feel like pathological “double counting”. If I make the calculator reliability Q and the subsequent duplication chance P, the behaviour here just seems unproblematically correct. (The fact that the 0.99 chance of cloning comes from i.i.d. experiments + law of large numbers in the Calculator Bet case and from a single randomisation device here should not matter.)
There is a separate point that EDT+(minimal reference class-SSA) is in general compatible with the ex ante/”UDT”-optimal policy and EDT+(anything that isn’t mrcSSA) is in general Dutch bookable. But that’s not what is going on here since mrcSSA does not mean “be updateless”/”use your prior in EU calculations”, it just means don’t anthropically update your prior based on how many observers are in different worlds.
[In the Calculator Bet case, if I additionally performed an SIA-update after seeing “X is true”, then I would be betting at 999999:1 odds, which would be very strange! But this is just the well-understood Dutch book against EDT+SIA from Briggs (2010).]
It seems to me that the only relevant difference here is the set of agents that you perceive yourself as being correlated with, and that the update being from empirical observation vs. logical reasoning is unimportant.
Replace “Y1” and “Y2″ with empirical statements which are probabilistically independent according to my prior, say, “the Hubble constant is x” and “the physical baryon density is y” and replace “prove a statement” with “conduct a definitive cosmological experiment”, and everything seems to go through the same. (If the thought is that we want all agents in a large world to verify the same statements, that’s dealt with by having the empirical statements be about global cosmological parameters.)
As noted in this comment, if you took yourself to control both agents that saw “X is true” and “X is false” in the original Calculator Bet, then you bet at 99:1 odds. And in your example, if you took yourself to only control ‘agents who proved Y1’ (rather than ‘agents who proved a statement’), then you bet at 1:1 odds.
Oh yeah, the Folk Theorem is totally consistent with the Nash equilibrium of the repeated game here being ‘everyone plays 30 forever’, since the payoff profile ‘-30 for everyone’ is feasible and individually-rational. In fact, this is the unique NE of the stage game and also the unique subgame-perfect NE of any finitely repeated version of the game.
To sustain ‘-30 for everyone forever’, I don’t even need a punishment for off-equilibrium deviations. The strategy for everyone can just be ‘unconditionally play 30 forever’ and there is no profitable unilateral deviation for anyone here.
The relevant Folk Theorem here just says that any feasible and individually-rational payoff profile in the stage game (i.e. setting dials at a given time) is a Nash equilibrium payoff profile in the infinitely repeated game. Here, that’s everything in the interval [-99.3, −30] for a given player. The theorem itself doesn’t really help constrain our expectations about which of the possible Nash equilibria will in fact be played in the game.
The ‘individual rationality condition’ is about the payoffs in equilibrium, not about the strategies. It says that the equilibrium payoff profile must yield to each player at least their minmax payoff. Here, the minmax payoff for a given player is −99.3 (which comes from the player best responding with 30 forever to everyone else setting their dials to 100 forever). The equilibrium payoff is −99 (which comes from everyone setting their dials to 99 forever). Since −99 > −99.3, the individual rationality condition of the Folk Theorem is satisfied.
Because the meaning of statements does not, in general, consist entirely in observations/anticipated experiences, and it makes sense for people to have various attitudes (centrally, beliefs and desires) towards propositions that refer to unobservable-in-principle things.
Accepting that beliefs should pay rent in anticipated experience does not mean accepting that the meaning of sentences are determined entirely by observables/anticipated experiences. We can have that the meanings of sentences are the propositions they express, and the truth-conditions of propositions are generally states-of-affairs-in-the-world and not just observations/anticipated experiences. Eliezer himself puts it nicely here: “The meaning of a statement is not the future experimental predictions that it brings about, nor isomorphic up to those predictions [...] you can have meaningful statements with no experimental consequences, for example: “Galaxies continue to exist after the expanding universe carries them over the horizon of observation from us.”″
As to how to choose one belief over another, if both beliefs are observationally equivalent in some sense, there are many such considerations. One is our best theories predict it: if our best cosmological theories predict something does not cease to exist the moment it exits our lightcone, then we should assign higher probability to the statement “objects continue to exist outside our lightcone” than the statement “objects vanish at the boundary of our lightcone”. Another is simplicity-based priors: the many-worlds interpretation of quantum mechanics is strictly simpler/has a shorter description length than the Copenhagen interpretation (Many-Worlds = wave function + Schrödinger evolution; Copenhagen interpretation = wave function + Schrödinger evolution + collapse postulate), so we should assign a higher prior to many-worlds than to Copenhagen.
If your concern is instead that attitudes towards such propositions have no behavioural implications and thus cannot in principle be elicited from agents, then the response is to point to the various decision-theoretic representation theorems available in the literature. Take the Jeffrey framework: as long as your preferences over propositions satisfies certain conditions (e.g. Ordering, Averaging), I can derive both a quantitative desirability measure and probability measure, characterising your desire and belief attitudes (respectively) towards the propositions you are considering. The actual procedure to elicit this preference relation looks like asking people to consider and compare actualising various propositions, which we can think of as gambles. For example, a gamble might look like “If the coin lands Heads, then one person comes into existence outside of our future lightcone and experiences bliss; If the coin lands Tails, then one person comes into existence outside of our future lightcone and experiences suffering”. Note, the propositions here can refer to unobservables. Also, it seems reasonable to prefer the above gamble involving a fair coin to the same gamble but with the coin biased towards Tails. Moreover, the procedure to elicit an agent’s attitudes to such propositions merely consists in the agents considering what they would do if they were choosing which of various propositions to bring about, and does not cash out in terms of observations/anticipated experiences.
(As an aside, doing acausal reasoning in general requires agent to have beliefs and desires towards unobservable-in-principle stuff in, e.g. distant parts of our universe, or other Everett branches).
Same as Sylvester, though my credence in consciousness-collapse interpretations of quantum mechanics has moved from 0.00001 to 0.000001.
Yeah great point, thanks. We tried but couldn’t really get a set-up where she just learns a phenomenal fact. If you have a way of having the only difference in the ‘Tails, Tuesday’ case be that Mary learns a phenomenal fact, we will edit it in!
Thanks, the clarification of UDT vs. “updateless” is helpful.
But now I’m a bit confused as to why you would still regard UDT as “EU maximisation, where the thing you’re choosing is policies”. If I have a preference ordering over lotteries that violates independence, the vNM theorem implies that I cannot be represented as maximising EU.
In fact, after reading Vladimir_Nesov’s comment, it doesn’t even seem fully accurate to view UDT taking in a preference ordering over lotteries. Here’s the way I’m thinking of UDT: your prior over possible worlds uniquely determines the probabilities of a single lottery L, and selecting a global policy is equivalent to choosing the outcomes of this lottery L. Now different UDT agents may prefer different lotteries, but this is in no sense expected utility maximisation. This is simply: some UDT agents think one lottery is the best, other might think another is the best. There is nothing in this story that resembles a cardinal utility function over outcomes that the agents are multiplying with their prior probabilities to maximise EU with respect to.
It seems that to get an EU representation of UDT, you need to impose coherence on the preference ordering over lotteries (i.e. over different prior distributions), but since UDT agents come with some fixed prior over worlds which is not updated, it’s not at all clear why rationality would demand coherence in your preference between lotteries (let alone coherence that satisfies independence).
Okay this is very clarifying, thanks!
If the preference ordering over lotteries violates independence, then it will not be representable as maximising EU with respect to the probabilities in the lotteries (by the vNM theorem). Do you think it’s a mistake then to think of UDT as “EU maximisation, where the thing you’re choosing is policies”? If so, I believe this is the most common way UDT is framed in LW discussions, and so this would be a pretty important point for you to make more visibly (unless you’ve already made this point before in a post, in which case I’d love to read it).
Yeah by “having a utility function” I just mean “being representable as trying to maximise expected utility”.
Ah okay, interesting. Do you think that updateless agents need not accept any separability axiom at all? And if not, what justifies using the EU framework for discussing UDT agents?
In many discussions on LW about UDT, it seems that a starting point is that agent is maximising some notion of expected utility, and the updatelessness comes in via the EU formula iterating over policies rather than actions. But if we give up on some separability axiom, it seems that this EU starting point is not warranted, since every major EU representation theorem needs some version of separability.
Don’t updateless agents with suitably coherent preferences still have utility functions?
Do you take the data in need of explanation to be a stream of orderly experiences, or one moment of orderly experience?