Stuart_Armstrong

Karma: 18,488

Stuart_Armstrong 10 Aug 2026 18:29 UTC
2 points
0
in reply to: idle-chat’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Simple Bayes means that, if there is no longer any anthropic angle to your problem, you should be able to apply non-anthropic updating.

Equivalently-ish ^[1] it implies that it doesn’t matter when you started thinking anthropically; if you started doing it yesterday and then updated on one day’s observation, it would be the same as if you started doing it today, or twenty years ago and updated on twenty years of observations. And it also doesn’t matter that you could have been X, if you know that you’re not X.

Formally, this is anchor-free: there isn’t an anchor moment that defines the beginning of when you’re thinking anthropically ^[2] .
1. ↩︎
  Anchor-free is stronger than simple Bayes; simple Bayes is just the equivalent of anchor-free in situations where there is no anthropic uncertainty. Anchor-free is the real condition I want, but simple Bayes is enough for the impossibility result.
2. ↩︎
  One needs to be careful here; technically, any method with a specified anchor is “anchor-free” in a certain sense if it says “whenever you actually started thinking anthropically, then go back to this moment in the past (or future) and anchor then”. If the specified anchor is part of the definition of the method, then it’s “anchor-free” in that it formally doesn’t matter when you started thinking anthropically, because in all cases you just go to a specified anchor ^[3] and role out everything backwards/forwards from that I don’t consider “anchor specified” to be anchor-free.
3. ↩︎
  “Go to a specified anchor” is not actually that trivial to define. If the “anchor specified” truly behaves like “anchor-free” then it has to work from the very moment of first consciousness (as that’s certainly a place you could start anchoring from), when you could have been, potentially, any being in the universe. So there’s a universal anthropic probability that depends potentially on the details of all agents who ever were or ever could be.

Stuart_Armstrong 10 Aug 2026 11:20 UTC
10 points
−5
on: FAQ: Isn’t AGI coming too soon for reprogenetics to help?
Society has a terrible track record with this kind of technology (see what liberal eugenics ended up looking like in practise—it wasn’t nazism, but it wasn’t anything positive, either).

Human intelligence is a very complicated system, playing out over the course of an individual’s life. All we have for the moment is evidence of small, local genetic effects. It would be a grave error to assume that these effects remain additive, without side-effects, when you combine them all. And measuring these side effects takes decades. Genetics isn’t like physics: you can’t go off-distribution (like sending a rocket to the moon) and assume the results hold up.

I think it wasn’t a coincidence that liberal eugenics went so wrong. The theory overpromised and was unsound, especially as you got into more details. It also played to people’s prejudices. This seems to be repeating the exact pattern—overpromising, unsound on the details, playing into people’s prejudices. Of course you don’t intend for things to go sour, and are advocating this for the best of motives—but the liberal eugenicists didn’t intend that, either, and good motives don’t guarantee good results.

At the very least, I’d want to see a detailed argument for why this won’t go wrong the way it’s gone wrong in the past. “This technology will only be used sensibly by sensible people under sensible governments” is not an assumption you get to make for free.

Stuart_Armstrong 9 Aug 2026 12:27 UTC
2 points
0
in reply to: idle-chat’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties

So on Sunday her epistemic state contains the statement “I am the original”, on Monday it does not.

Epistemic states can change freely without breaking martingales; on Sunday, she’s in room 1; put her to sleep and wake her up in room 1 or 2, randomly chosen; no duplications. Her epistemic state has moved from “I’m in room 1” to “I don’t know what room I’m in”, and yet the martingale works perfectly across this.

In any case, the statement is not “I am the original” but “on Monday, I was the original”, which is preserved.

I’ve also weakened the requirements; now it’s the law of total probability that is violated, not the martingale.

If you get , you’re violating simple Bayes on the initial Sunday, which forces Q to equal the prior on worlds.

Stuart_Armstrong 9 Aug 2026 12:21 UTC
2 points
0
in reply to: JBlack’s comment on: Simplifying the anthropic impossibility result
Yep, seems right.

Personally, I consider false memory to be a form of anthropic scenarios—you don’t know exactly who you are.

Stuart_Armstrong 8 Aug 2026 21:05 UTC
6 points
0
in reply to: jessicata’s comment on: Simplifying the anthropic impossibility result
Yep!

I suppose it’s back to my old conclusion—decision theory is more fundamental that probability theory. https://arxiv.org/abs/1110.6437

Stuart_Armstrong 8 Aug 2026 18:06 UTC
2 points
0
in reply to: jessicata’s comment on: Simplifying the anthropic impossibility result
That’s standard Sleeping Beauty; I crafted duplicate Sleeping Beauty so that there was no memory-loss issue. Sleeping Beauty always knows what day it is.

Are you saying that SIA+SSSA suffers from these problems?

Stuart_Armstrong 8 Aug 2026 17:53 UTC
4 points
0
in reply to: jessicata’s comment on: Simplifying the anthropic impossibility result
SIA is well defined, so you can calculate with it, and notice the probability of Heads go from ¹⁄₂ on Sunday to ¹⁄₃ on Monday to 0 or ¹⁄₂ on Tuesday. The Monday → Tuesday transition is consistent with probability rules, the Sunday → Monday is not. Inclusion-exclusion (now replaced with the law of total probability) is one of the easiest ways to pinpoint the inconsistency.

SSA has multiple definitions, but in it’s “halfer” format, with reference class being Sleeping Beauty copies currently in that universe, it assigns ¹⁄₂ to Heads on Sunday, ¹⁄₂ on Monday, and ²⁄₃ or 0 on Tuesday. It is Bayes consistent, but violates simple Bayes on Tuesday. Other SSAs may behave differently.

It assign probabilities for the agents inside to know their centered world (that is, all physical facts plus “what is here + now”), that’s enough for things like “estimating size of universe” or “figuring out if today is Tuesday” or “figuring out if coin is heads”, which people disagree on.

I feel that it probably has timeline inconsistencies, such as assigning a different size to the universe at different moments, without any observations to update on. The source of this feeling is that if it were consistent on timelines, then it would assign a reasonable internal agent distribution to future observations.

After all, if you’ve assigned probabilities to the day, the coin, how many agents there are, and the room, this should be now constraining your expected observations.

Stuart_Armstrong 8 Aug 2026 17:23 UTC
4 points
0
in reply to: jessicata’s comment on: Simplifying the anthropic impossibility result
That design is SIA over full histories (see the end of the post), which breaks simple Bayes on Sunday.

The simplest version of SIA (C-SIA, counting SIA) upweighs worlds by the number of copies of you there is in them at any one time and is otherwise fully Bayesian. This is consistent with deaths, divergence of copies, and the initial creation of you(s). The impossibility result shows that it isn’t consistent with duplication, though.

A potential criticism of SSSA+SIA: if it doesn’t intrinsically assign subjective probabilities to agents inside the problem, what is it for? These problems aren’t difficult to analyse probabilistically from the outside; it’s from the inside that we need an effective tool.

Stuart_Armstrong 8 Aug 2026 12:10 UTC
4 points
0
in reply to: jessicata’s comment on: Simplifying the anthropic impossibility result
What does this give as P(Monday|Sunday)? Seems it has to be zero.

Now obviously P(Monday|Sunday)=0 in the sense that “If it’s Sunday, it’s not Monday”, but how does SSSA+SIA encode “If it’s Sunday, what’s the probability that it will be Monday?” or “If I’ve observed Sunday, what’s the probability I will observe Monday?” It feels that it’s given up causality.

Unless I’ve made a mistake (very possible), my result shows that if SSSA+SIA can be turned into a personal probability distribution for Sleeping Beauty to use for predicting her experiences as well as world-facts, it will then break ^[1] in that form.

EDIT: I’ve replaced inclusion-exclusion with the law of total probability and simplified the proof further.
1. ↩︎
  The fundamental reason it will break is that h_1 and h_2 are mutually exclusive and exhaustive full histories for Sleeping Beauty, but any sensible probability system wants to put P(h_1)=1 and P(h_2)=1/2 (which are the correct probabilities for the existence of these histories, since they are not mutually exclusive from the outside).
  
  The system has too few degrees of freedom; you can’t take the outside-observer probability distribution (which is what simple Bayes forces on Sunday and Tuesday), substitute in mutual exclusivity, and not have something break. Inclusion-exclusion is just the simplest thing that shows the break, currently.

Stuart_Armstrong 8 Aug 2026 11:35 UTC
2 points
0
in reply to: idle-chat’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Simpler version: https://www.lesswrong.com/posts/mi5YYtsdbioHEYxiY/simplifying-the-anthropic-impossibility-result

Simplifying the anthropic impossibility result

Stuart_Armstrong8 Aug 2026 8:25 UTC

23 points

13 comments3 min readLW link

Stuart_Armstrong 7 Aug 2026 16:42 UTC
2 points
0
in reply to: idle-chat’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Yes, I did mean ; thanks, corrected.

Will link to a new post that explains the impossibility result better.

Stuart_Armstrong 7 Aug 2026 15:58 UTC
2 points
0
in reply to: Swaraag’s comment on: Value generalisation: value correction
That is very interesting. Thanks for continuing the work.

So it seems that, in this instance, even though R_c fixed the behaviour, it wasn’t by finding the correct feature but by filtering out the misbehaviour (giant frowny face). It seems that we may need to use the full machinery of ACE (see here and here) - or maybe even that wouldn’t be enough, with such sparse data?

Have sent you an email.

Stuart_Armstrong 6 Aug 2026 6:16 UTC
2 points
0
in reply to: JBlack’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
I agree about 75% with this perspective.

Stuart_Armstrong 5 Aug 2026 16:21 UTC
4 points
0
in reply to: tgb’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Ok, got the issue. Thanks!

The condition I’m calling “simple Bayes” is that, if there is no doubt as to which agent you are in the world, then proceed by taking the prior over worlds and updating on the evidence “there exists a person in this world who have made this observation” (in non-anthropic situations, “there exists a person who has observed history H” and “I have observed history H” contain the same information).

On Monday, there is uncertainty are to which agent SB is. On Tuesday there is not: the two agents in the Tails world can tell each other apart, based on room number observation.

So simple Bayes applies to Sunday and Tuesday, but not to Monday.

On Sunday, the observations are independent of the coin flip, so P(“I saw Sunday” | w_T) = P(“I saw Sunday” | w_H) = 1.

If h_0 = (“I saw Sunday”), then P(w_T | h_0) = P(h_0 | w_T)P(w_T)/((P(h_0 | w_T)P(w_T) + P(h_0 | w_H)P(w_H)) = 1*(1/2)/(1/2+1/2)=1/2. Same thing for P(w_H | h_0).

On Tuesday, in the tails world, there exists, with certainty, an agent who has observed: h_1=(“it’s Sunday”, “I’m awake on Monday”, “it’s Tuesday and my room number is 1”). Similarly, in the heads world, there exists, with certainty, an agent (the only agent) with the same observation.

Since both exist with certainty, then the formula is the same as before: P(w_T | h_1) = P(h_1 | w_T)P(w_T)/((P(h_1 | w_T)P(w_T) + P(h_1 | w_H)P(w_H)) = 1*(1/2)/(1/2+1/2)=1/2, and the same for P(w_H | h_1).

So the naive direct application of Bayes in non-anthropic situations gives us these values. The impossibility result is just that, given these values for Sunday and Tuesday, there are no values on Monday (the time-slice where we can’t use simple Bayes because there is genuine uncertainty as to which agent SB is) that allow the martingale condition to extend between Sunday and Tuesday.

Note that I’m not saying that SIA or SSA are wrong or that you can’t do anthropic probability. I’m saying that if you do do anthropic probability, you have to drop some intuitive properties possessed by standard probability.

For instance SIA drops the martingale condition from Sunday to Monday (indeed SIA always obeys simple Bayes). SSA (which is less uniquely defined) either drops the martingale condition from Monday to Tuesday or drops simple Bayes on Tuesday (using “centered worlds” is an explicit acknowledgement of dropping simple Bayes).

Stuart_Armstrong 5 Aug 2026 14:09 UTC
2 points
0
in reply to: tgb’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
To get the values of Q(T| awake, room 2) and Q(T| awake, room 1), I am applying “simple Bayes” because upon seeing awake and room 1, there is no longer any anthropic uncertainty as to who you are in the world.

Then, formally, I am assuming that Q is a well defined probability in general (obeying all the Bayesian equations, amongst others) and aiming to get a contradiction from this assumption.

So I set up an equation involving terms like Q(room 1 | awake) in order to infer what these would have to be. I’ve used simple Bayes to pin down some of the terms, and then I’m using the assumption that Q is well defined to set up an equation between the known terms and the unknown terms.

So one can write:

Q(T| awake, room 1) = (1/2) (t) / (1-t/2) = t/(2 - t)

And then, plugging in the known value of Q(T | awake, room 1), one deduces Q(T | awake) = ²⁄₃. I haven’t assumed that value; I’ve deduced it, under the “Q is reasonable, martingale, and simple Bayes works” assumptions (actually, all that I can deduce—and all that I need—is that Q(T | awake) > ¹⁄₂; getting it to be ²⁄₃ requires slightly stronger symmetry assumptions).

I can also deduce Q(T | awake) = ¹⁄₂ (martingale from Sunday to Monday), and thus get a contradiction. One of my assumptions must be wrong. Q cannot simultaneously obey Bayesian updating, obey the martingale, and obey simple Bayes.

Stuart_Armstrong 5 Aug 2026 13:58 UTC
2 points
0
in reply to: avturchin’s comment on: III. Anthropic reasoning has issues with infinite worlds; D-SIA can fix this
Yep.

I’m wondering if the finite speed of light + planks constant giving a minimum coarsness to the universe + the cosmological constant giving a maximal size to the universe + the landauer limit preventing infinite computations in a finite universe… might not be strong evidence for a non-infinite universe?

Stuart_Armstrong 5 Aug 2026 13:52 UTC
4 points
0
in reply to: JBlack’s comment on: II. Anthropic reasoning with duplication is not consistent with the usual probability properties
Yep, the martingale is satisfied for both.

It’s the equivalence between (conditional duplication) and (unconditional duplication + conditional killing) that I’m disputing (or at least saying that you can’t just assume for free).

The difference is clear for the formal martingale: let’s count the full agent histories from beginning to end. Conditional duplication has three histories - (Sunday, awake, room 1, T), (Sunday, awake, room 2, T) and (Sunday, awake, room 1, H). While (unconditional duplication + conditional killing) has four—those three plus extra one heads: the “death” history of just (Sunday).

Thus the martingale in the (ud + ck) case has an extra possible future observation to consider: specifically, the non-observation. That’s why “awake” gives information in the (ud+ck) case (not all your duplicates would be guaranteed to see it) but not in the (cd) case (where all your duplicates are guaranteed to see it).

(It doesn’t help if one says that “non-observations don’t count”, because then the theory breaks breaks the martingale in the case of just “conditional killing” on its own.)

Now, philosophically, I’m partial to the idea that “instant duplication followed by deletion” should be the same as “not creating duplicates at all”; that’s a strong pro-SIA argument. But that doesn’t fix the martingale; it instead shows that if we assumed that there were instantly-deleted duplicates, then that assumption would fix the martingale.

Stuart_Armstrong 4 Aug 2026 17:09 UTC
2 points
0
in reply to: avturchin’s comment on: Sleeping Beauty as a Mind Killer
But what is the mechanism by which the doomsday argument works? The Laplace example has a clear prior powering it, but why should it apply here?

Stuart_Armstrong 4 Aug 2026 15:07 UTC
4 points
0
in reply to: avturchin’s comment on: Anthropic reasoning without duplicates is just standard Bayesian updating
The alternative I found, D-SIA, does extend to infinite universes but isn’t strongly fond of them (it likes large worlds, but doesn’t have an infinite update towards infinite worlds).

Stuart_Armstrong

Sim­plify­ing the an­thropic im­pos­si­bil­ity result

Simplifying the anthropic impossibility result