Glad to hear you’re planning to write up a post covering stuff like this! I personally think it’s quite overdue, especially on a site like this which I suspect has an inherent selection effect on people who take ideas quite seriously like me. I don’t quite understand the last part of your reply though, I understand the importance of measure in decision making but like I said in my post, I thought if the blackmailer makes a significant number of simulations then indexical uncertainty could still be established since it could still have a significant effect on your future observer moments. Did I make a mistake anywhere in my reasoning?
sisyphus
Karma: −11
[Question] What does “probability” really mean?
[Question] A thought experiment
Hi, thanks for your reply and for approving my post. I definitely get what you mean when you said “people trapped in a kinda anxiety loop about acausal blackmail”, and admittedly I do consider myself somewhat in that category. However, simply being aware of this doesn’t really help me get over my fears, since I am someone that really likes to hear concrete arguments about why stuff like this doesn’t work instead of just being satisfied with a simple answer. You said that you had to deal with this sort of thing a lot so I presume you’ve heard a bunch of arguments and scenarios like this, do you mind sharing the reasons why you do not worry about it?
[Question] How does acausal trade work in a deterministic multiverse?
[Question] Can we, in principle, know the measure of counterfactual quantum branches?
I disagree with your 300 room argument. My identity is tied to my mind, which is a computation carried out by all 300 copies of my body in these 300 rooms. If all 300 rooms were suddenly filled with sleeping gas and 299 of the copies are quietly killed, only 1 copy will wake up. However, I should expect to experience waking up with probability 1 since that is the only possible next observer moment in this set up. The 299 dead copies of me cannot generate any further observer moments since they are dead.
I’d argue that you cannot experience a coma since you’re unconscious, you can’t really experience anything when you’re in a coma. When you wake up it will feels as if you have just jumped forward in time. Deep sleep is qualitatively different and I was careful to avoid saying it is exactly like sleep, since sleep is simply a state of lowered consciousness instead of complete unconsciousness (e.g. we can still experience things like dreams). It is possible coma is also lowered consciousness like sleep, in which case you can experience comas but this says nothing about experiencing unconsciousness.
I think the main crux here is the philosophy of identity. You do not regard the emulated mind running on a machine on the other side of the room as “you”, but if the subjective experiences are identical, you cannot rule out the possibility of being the emulated mind yourself. They are functionally identical and thus should be both considered “you” as they are carrying out the same computation.
“And to be consistent one would be adviced to “expect” at any moment to fluctuate into mars and choke for lack of air.”
You’re right that this is a probability and in a multiverse this possibility would be realised. But in this case the probability of it occurring is so little we needn’t pay attention to it. This is qualitatively different from “low probability of being resurrected”, because “teleporting to Mars” has a counterfactual experience of “not teleporting to Mars” which outweighs its probability, whereas “experiencing resurrection” does not have a corresponding counterfactual experience of “experiencing non-experience” since this is a tautologically meaningless statement.
“And it is not like the universe would suddenly abandon its indifference to human desire just for respecting non-destruction.”
I am not saying the universe necessarily has to care about human experiences at all, just that death and nonexperience is a tautologically meaningless concept. In fact I’d rather the universe did not make me immortal.
“You seem to be arguing that you will experience “yourself” in many other parts of a multiverse after you die. Why does this not occur before you die?”
Because even though “you” in the sense of a computation have multiple embeddings in the multiverse, the vast vast majority of them share the same subjective experience and are hence functionally the same being, you can’t yourself distinguish between them. The difference is that while some of these embeddings end when they die, you will only experience the ones which continue on afterwards (since you can’t experience nonexperience).
“Can you clarify what you mean by “feel like falling asleep”? I don’t see why there would be even a moment of unconsciousness if your idea is true.”
I meant that death just feels like a timeskip subjectively. And you’re correct that there wouldn’t be any moments of unconsciousness, in fact this is already true in real life. You can’t experience unconsciousness, and when you wake up it subjectively feels like a jump-forward from where you left off.
Yea my main question is that can we even in principle estimate the pure measure of existence of branches which diverged from our current branch? We can know the probabilities conditioned on the present but I don’t see how we can work backwards to estimate the probabilities of a past event not occurring. Just like how a wavefunction can be evolved forward after a measurement but cannot be evolved backwards from the measurement itself to deduce the probability of obtaining such a measurement. Or can we?
I mainly picked “world where WW2 did not happen” to illustrate what I mean by counterfactual branches, in the sense that it has already diverged from us and is not in our future.
Woops, edited. Thanks! :)
Ah I see. Sorry for not being too familiar with the lingo but does uniform prior just mean equal probability assigned to each possible embedding?
I suppose the lollipops are indeed an unnecessary addition, so the final question can really be reframed as “what is the probability that you will see heads?”
Right, so your perspective is that due to the multiple embeddings of yourself being in the heads scenario, it is the 1001:1 option. That line of reasoning is kind of what I thought as well, but it was against the 1:1 odds as would be suggested by my intuition. I guess this is the same as the halfer vs thirder debate, where 1:1 is the halfer position and the 1001:1 is the thirder position.
I see, thanks. IMO the two are indeed quite similar but I think my example illustrates the problem of self-location uncertainty in a clearer way. That being said, what is your thought on the probability of getting a lollipop if you’re in such a scenario? Are the odds 1:1 or 1001:1?
Like I said, “what they want” is irrelevant to the discussion here, you can imagine them wanting virtually anything. The danger lies in understanding the mechanism. You can imagine the alien telling you to order a chocolate ice cream instead of vanilla because that somehow via the butterfly effect yields positive expected utility for them (e.g. by triggering a chain of subtle causal events that makes the AGI we build slightly more aligned with their values or whatever). The problem is that there will also be an alien that wants you to order vanilla instead of chocolate, and who is also fine with applying a negative incentive. Sure, this means you can order whatever flavor of ice cream you want since you will get punished either way, but you’re still getting punished (not good).
The point is “what it wants [us] to do” can essentially be anything we can imagine thanks to the many-gods “refutation” where every possible demand can be imposed on us by some alien on some branch of the quantum multiverse. It can be as ridiculous as leaving your front door open on a Wednesday night or flushing away a straw down a toilet at 3 am, whatever eventually leads to more positive utility to the blackmailer via the butterfly effect (e.g. maybe flushing that straw down the toilet leads to a chain of causal events which makes the utility function of the AGI we build in the future to be slightly more aligned with their goals). “What the alien wants” is irrelevant here, the point is that now you know the mechanism by which aliens can coerce you into doing what they want, and merely knowing so gives other agents increased incentive to acausally extort you. You seem to be hung up on what exactly I’m scared the blackmailer wants me to do, what I am actually worried about is that simply knowing the mechanism imposes danger. The real basilisk is the concept of acausal extortion itself because it opens us up to many dangerous scenarios, not that I am worried about any specific scenario.
The reason why we cannot acausally trade with artificial superintelligences is because we lack the computing power to simulate them accurately, so ASIs would not have any incentive to actually commit to cooperate in a prisoner’s dilemma style situation instead of just letting us believe it will while it secretly defects. But we don’t have this same problem with non-superintelligences like aliens or even humans who have succeeded in aligning their own AIs, since we can actually simulate such beings in our head. What I am looking for is a concrete argument against this possibility.
The point is that X can essentially be any action, for the sake of the discussion let’s say the alien wants you to build an AGI that maximizes the utility function of the alien in our branch of the multiverse.
My main point is that the many-gods refutation is a refutation against taking a specific action, but is not a refutation against the fact that knowing about acausal extortion increases the proportion of bad future observer moments. It in fact makes it worse because, well, now you’ll be tortured no matter what you do.
I don’t think this would help considering my utter lack of capability to carry out such threats. Are there any logical mistakes in my previous reply or in my concerns regarding the usual refutations as stated in the question? I’ve yet to hear anyone engage with my points against the usual refutations.
Completely agree here. I’ve known the risks involved for a long time, but I’ve only really felt them recently. I think Robert Miles phrases it quite nicely on the Inside View podcast, where “our System 1 thinking finally caught up with our System 2 thinking.”