Stuart_Armstrong

Karma: 18,431

Stuart_Armstrong 6 Aug 2026 6:16 UTC
2 points
0
in reply to: JBlack’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
I agree about 75% with this perspective.

Stuart_Armstrong 5 Aug 2026 16:21 UTC
4 points
0
in reply to: tgb’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Ok, got the issue. Thanks!

The condition I’m calling “simple Bayes” is that, if there is no doubt as to which agent you are in the world, then proceed by taking the prior over worlds and updating on the evidence “there exists a person in this world who have made this observation” (in non-anthropic situations, “there exists a person who has observed history H” and “I have observed history H” contain the same information).

On Monday, there is uncertainty are to which agent SB is. On Tuesday there is not: the two agents in the Tails world can tell each other apart, based on room number observation.

So simple Bayes applies to Sunday and Tuesday, but not to Monday.

On Sunday, the observations are independent of the coin flip, so P(“I saw Sunday” | w_T) = P(“I saw Sunday” | w_H) = 1.

If h_0 = (“I saw Sunday”), then P(w_T | h_0) = P(h_0 | w_T)P(w_T)/((P(h_0 | w_T)P(w_T) + P(h_0 | w_H)P(w_H)) = 1*(1/2)/(1/2+1/2)=1/2. Same thing for P(w_H | h_0).

On Tuesday, in the tails world, there exists, with certainty, an agent who has observed: h_1=(“it’s Sunday”, “I’m awake on Monday”, “it’s Tuesday and my room number is 1”). Similarly, in the heads world, there exists, with certainty, an agent (the only agent) with the same observation.

Since both exist with certainty, then the formula is the same as before: P(w_T | h_1) = P(h_1 | w_T)P(w_T)/((P(h_1 | w_T)P(w_T) + P(h_1 | w_H)P(w_H)) = 1*(1/2)/(1/2+1/2)=1/2, and the same for P(w_H | h_1).

So the naive direct application of Bayes in non-anthropic situations gives us these values. The impossibility result is just that, given these values for Sunday and Tuesday, there are no values on Monday (the time-slice where we can’t use simple Bayes because there is genuine uncertainty as to which agent SB is) that allow the martingale condition to extend between Sunday and Tuesday.

Note that I’m not saying that SIA or SSA are wrong or that you can’t do anthropic probability. I’m saying that if you do do anthropic probability, you have to drop some intuitive properties possessed by standard probability.

For instance SIA drops the martingale condition from Sunday to Monday (indeed SIA always obeys simple Bayes). SSA (which is less uniquely defined) either drops the martingale condition from Monday to Tuesday or drops simple Bayes on Tuesday (using “centered worlds” is an explicit acknowledgement of dropping simple Bayes).

Stuart_Armstrong 5 Aug 2026 14:09 UTC
2 points
0
in reply to: tgb’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
To get the values of Q(T| awake, room 2) and Q(T| awake, room 1), I am applying “simple Bayes” because upon seeing awake and room 1, there is no longer any anthropic uncertainty as to who you are in the world.

Then, formally, I am assuming that Q is a well defined probability in general (obeying all the Bayesian equations, amongst others) and aiming to get a contradiction from this assumption.

So I set up an equation involving terms like Q(room 1 | awake) in order to infer what these would have to be. I’ve used simple Bayes to pin down some of the terms, and then I’m using the assumption that Q is well defined to set up an equation between the known terms and the unknown terms.

So one can write:

Q(T| awake, room 1) = (1/2) (t) / (1-t/2) = t/(2 - t)

And then, plugging in the known value of Q(T | awake, room 1), one deduces Q(T | awake) = ²⁄₃. I haven’t assumed that value; I’ve deduced it, under the “Q is reasonable, martingale, and simple Bayes works” assumptions (actually, all that I can deduce—and all that I need—is that Q(T | awake) > ¹⁄₂; getting it to be ²⁄₃ requires slightly stronger symmetry assumptions).

I can also deduce Q(T | awake) = ¹⁄₂ (martingale from Sunday to Monday), and thus get a contradiction. One of my assumptions must be wrong. Q cannot simultaneously obey Bayesian updating, obey the martingale, and obey simple Bayes.

Stuart_Armstrong 5 Aug 2026 13:58 UTC
2 points
0
in reply to: avturchin’s comment on: III. Anthropic reasoning has issues with infinite worlds; D-SIA can fix this
Yep.

I’m wondering if the finite speed of light + planks constant giving a minimum coarsness to the universe + the cosmological constant giving a maximal size to the universe + the landauer limit preventing infinite computations in a finite universe… might not be strong evidence for a non-infinite universe?

Stuart_Armstrong 5 Aug 2026 13:52 UTC
4 points
0
in reply to: JBlack’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Yep, the martingale is satisfied for both.

It’s the equivalence between (conditional duplication) and (unconditional duplication + conditional killing) that I’m disputing (or at least saying that you can’t just assume for free).

The difference is clear for the formal martingale: let’s count the full agent histories from beginning to end. Conditional duplication has three histories - (Sunday, awake, room 1, T), (Sunday, awake, room 2, T) and (Sunday, awake, room 1, H). While (unconditional duplication + conditional killing) has four—those three plus extra one heads: the “death” history of just (Sunday).

Thus the martingale in the (ud + ck) case has an extra possible future observation to consider: specifically, the non-observation. That’s why “awake” gives information in the (ud+ck) case (not all your duplicates would be guaranteed to see it) but not in the (cd) case (where all your duplicates are guaranteed to see it).

(It doesn’t help if one says that “non-observations don’t count”, because then the theory breaks breaks the martingale in the case of just “conditional killing” on its own.)

Now, philosophically, I’m partial to the idea that “instant duplication followed by deletion” should be the same as “not creating duplicates at all”; that’s a strong pro-SIA argument. But that doesn’t fix the martingale; it instead shows that if we assumed that there were instantly-deleted duplicates, then that assumption would fix the martingale.

Stuart_Armstrong 4 Aug 2026 17:09 UTC
2 points
0
in reply to: avturchin’s comment on: Sleeping Beauty as a Mind Killer
But what is the mechanism by which the doomsday argument works? The Laplace example has a clear prior powering it, but why should it apply here?

Stuart_Armstrong 4 Aug 2026 15:07 UTC
4 points
0
in reply to: avturchin’s comment on: Anthropic reasoning without duplicates is just standard Bayesian updating
The alternative I found, D-SIA, does extend to infinite universes but isn’t strongly fond of them (it likes large worlds, but doesn’t have an infinite update towards infinite worlds).

Stuart_Armstrong 4 Aug 2026 15:06 UTC
2 points
0
in reply to: Noosphere89’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Yep, roughly where I’m at.

Stuart_Armstrong 4 Aug 2026 15:04 UTC
2 points
0
in reply to: avturchin’s comment on: Sleeping Beauty as a Mind Killer
Laplace’s result comes from a uniform prior over the probability of the sun not rising, updated on observations. It’s sort of the maximally uniformative prior.

However, the Doomsday argument is an extra update on top of evidence that we’ve already seen. If we know the priors, we don’t have a doomsday effect. So unless we genuinely do think that each year there’s an independent probability of doom, draw with fixed but unknown probability p, Laplace’s sunrise problem and the doomsday argument shouldn’t apply, right?

Stuart_Armstrong 4 Aug 2026 9:31 UTC
2 points
0
in reply to: JBlack’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
SIA (which I tend to agree with) treats unconditional and conditional duplication as the same; but that’s an assumption we don’t get to take for free.

On heads, being killed or not being asked have clear effects in terms of expectation of alternative observations to see or not see.

That’s why unconditional duplication is fully martingale, while conditional duplication isn’t.

Stuart_Armstrong 4 Aug 2026 9:23 UTC
2 points
0
in reply to: JBlack’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Ok! So you agree with the impossibility result, just disagree that martingale is a reasonable property in the first place.

That’s fine; just be aware of what you might be giving up. Which is that now you must update on entirely predicted observations. e.g. if duplication can happen in the middle of a conscious experience (so no sleeping required) then you might know that, for instance, you will update in the middle of an exercise routine.

Stuart_Armstrong 4 Aug 2026 9:14 UTC
4 points
0
in reply to: jessicata’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Yep, I fully agree :-) I’ve already said that decision theory should prime over probability theory; examples like this are why.

Any probability theory is incomplete, though, because if there’s a Dutch book where the people paying and receiving are arguably different people, the Dutch books outcome shifts depending on how they value each other. So Sleeping Beauty who is an average utilitarian over her future copies will behave like SSA has the true probabilities (for selfish bets).

Stuart_Armstrong 3 Aug 2026 23:23 UTC
3 points
0
on: Sleeping Beauty as a Mind Killer
Gott’s version is based on the idea that I am located near the middle of the total sequence of observers and does not explicitly compare short and long possible worlds.
As I argued here in the “Edge Of The Lake argument”, this type of reasoning seems clearly wrong in other contexts, so why should it be true in anthropic reasoning?

Stuart_Armstrong 3 Aug 2026 23:20 UTC
4 points
0
in reply to: avturchin’s comment on: Anthropic reasoning without duplicates is just standard Bayesian updating
As I argued here, C-SIA (counting SIA) argues for infinite numbers of observers and then breaks in worlds with infinite numbers of observers.

That seems a reducio against C-SIA, not an argument for infinite universes!

(D-SIA, the alternative I’m arguing for, deals fine with infinite universes, but involves some more choices on our parts).

Stuart_Armstrong 3 Aug 2026 21:41 UTC
2 points
−2
in reply to: ike’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Predicted observation. You know you’ll be observing waking up on Monday (or, at least, that every thread going on from you to will observe waking up on Monday). And yet you are not, now, currently in the epistemic state you will be on Monday, despite know exactly what you will observe.

Bbtw, death and birth/creating are opposites, and neither is really a problem in anthropic probability. The opposite of duplication is merger—much more difficult to achieve, but, yes, it has similar issues to duplication.

“The thread of my experience has forked/merged” is something probability theories were not built to deal with; unlike threads of experience ending and starting, which is can deal with fine.

Stuart_Armstrong 3 Aug 2026 21:36 UTC
2 points
0
in reply to: tgb’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
I am applying a theory of probability that I’m taking as reasonable until I reach a contradiction that shows that it isn’t (proof by contradiction). If P is reasonable, it should give a value to statements like P(T | awake), and that value should follow the normal Bayes rules.

I don’t think I ever injected a specific value that wasn’t already there in the assumptions?

Stuart_Armstrong 3 Aug 2026 20:11 UTC
4 points
0
in reply to: tgb’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
No, that’s just Bayes, conditional on another factor, awake (I should have written it last for higher clarity).

Bayes is: P(A|B)=P(B|A)*P(A)/P(B)

Bayes with an extra condition is: P(A|B,C)=P(B|A,C)*P(A|C)/P(B|C)

(or you can define Q(X)=P(X|C) and do standard Bayes with Q(X))

Stuart_Armstrong 3 Aug 2026 20:07 UTC
2 points
−2
in reply to: ike’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties

If I can magically increase my probability in outcome A by committing to suicide in not-A worlds

You can’t. You can increase your relative probability of seeing outcome A, conditional on you being alive. You split into: see A, see not A, be dead. Both of the later are in world not A, and you can shift their relative size, but not their total size.

It’s survivorship bias via suicide, nothing strange about it.

Stuart_Armstrong 3 Aug 2026 16:47 UTC
2 points
0
in reply to: ike’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Sorry, the latex display should be fixed now.

She doesn’t have a probability on a particular room or of a coin result as of Monday because she doesn’t observe that on Monday.

Pertinent comment, so I’ve added an of observing the coin.

I’ve got a different scenario with no duplicates. A fair coin is flipped today, if it’s heads, Roulette Rory gets killed with 75% probability before learning what the coin says.

In this case, not making an observation (due to death) is another possible history. The possible future histories are “see heads”, “see tails”, and “{}” (see the other post on how deaths don’t affect Bayesian probability).

That’s also the reason why ID(Sleeping Beauty) doesn’t have the paradox; the duplicates existing but not observing their history means that heads Sleeping Beauty gains information on Monday (I’m not dead, hence I wasn’t the duplicate that died).

Stuart_Armstrong 3 Aug 2026 12:41 UTC
2 points
0
in reply to: StanislavKrym’s comment on: Value Generalisation 2: The Missing Hole in AIs’ abilities
I believe that part of our out-of-distribution and reasoning abilities are unavailable to our conscious thought. But they don’t seem to be like image recognition neural nets (grasp the pretrained concept) nor LLMs (complete the pattern), nor any variant I know of.

But there’s something structural going on in there (Klein’s work is a small example pointing to this), and it’s this structure I intend to use.