This is not the same as CEV. CEV involves the AI extrapolating a user’s idealized future values and acting to implement them, even overriding current preferences if needed, whereas my model forbids that. In my framework, the AI never drives or predicts value change; it simply provides accurate world models and optimal plans based on the user’s current values, which only the user can update.
CEV also assumes convergence; my model protects normative autonomy and allows value diversity to persist.
Of course it is, but I’m a functionalist
I’d like to offer a counterargument, that, I’ll admit, can get into some pretty gnarly philosophical territory quite quickly.
Premise 1: We are not simulated minds—we are real, biological observers.
Premise 2: We can treat ourselves as a random sample drawn from the set of all conscious minds, with each mind weighted by some measure—i.e., a way of assigning significance or “probability” to different observers. The exact nature of this measure is still debated in cosmology and philosophy of mind.
Inference: If we really are a typical observer (as Premise 2 assumes), and yet we are not simulated (as Premise 1 asserts), then the measure must assign significantly greater weight to real biological observers than to simulated ones. This must be true even if there are vastly more simulations in a numerical sense—even uncountably infinitely more—because our non-simulated status would be extremely improbable otherwise.
Conclusion: So, under the assumption that we are typical, our existence as real observers implies that simulated minds must have much lower measure than real ones. Therefore, even if digital minds exist in large numbers, they may not matter proportionally in ethical calculations—since their measure, not just their count, determines their relevance. This gives us reason to think utilitarianism, when properly weighted by measure, may still prioritize the welfare of real, biological minds.
This sounds like another crazy thing that the logic says is right but is probably not right, but I don’t know why.
Also, does this imply that a technologically mature civilization can plausibly create uncountably infinite conscious minds? What about other sizes of infinity? This, I suppose, could have weird implications for the measure problem in cosmology.
I’m not sure if I understand, but sounds interesting. If true, does this have any implications for ethics more broadly, or are the implications confined only to our interpretation of computations?
Maybe. But what do you mean by, “you can narrow nothing down other than pure logic”?
I interpret the first part—”you can narrow nothing down”—to mean that the simulation argument doesn’t help us make sense of reality. But I don’t understand the second part: “other than pure logic.” Can you please clarify this statement?
Thank you, I feel inclined to accept that for now.
But I’m still not sure, and I’ll have to think more about this response at some point.
Edit: I’m still on board with what you’re generally saying, but I feel skeptical of one claim:
It seems to me the main ones produce us via base physics, and then because there was an instance in base physics, we also get produced in neighboring civilizations’ simulations of what other things base physics might have done in nearby galaxies so as to predict what kind of superintelligent aliens they might be negotiating with before they meet each other.
My intuition tells me there will probably be superior methods of gathering information about superintelligent aliens. To me, it seems like the most obvious reason to create sims would be to respect the past for some bizarre ethical reason, or for some weird kind of entertainment, or even to allow future aliens to temporarily live in a more primitive body. Or perhaps for a reason we have yet to understand.
I don’t think any of these scenarios would really change the crux of your argument, but still, can you please justify your claim for my curiosity?
I think I understand your point. I agree with you: the simulation argument relies on the assumption that physics and logic are the same inside and outside the simulation. In my eyes, that means we may either accept the argument’s conclusion or discard that assumption. I’m open to either. You seem to be, too—at least at first. Yet, you immediately avoid discarding the assumption for practical reasons:
If we have no grasp on anything outside our virtualized reality, all is lost.
I agree with this statement, and that’s my fear. However, you don’t seem to be bothered by the fact. Why not? The strangest thing is that I think you agree with my claim: “The simulation argument should increase our credence that our entire understanding of everything is flawed.” Yet somehow, that doesn’t frighten you. What do you see that I don’t see? Practical concerns don’t change the territory outside our false world.
Second:
It seems to me the main reason is because we’re near a point of high influence in original reality and they want to know what happened—the simulations then are effectively extremely high resolution memories.
That’s surely possible, but I can imagine hundreds of other stories. In most of those stories, altruism from within the simulation has no effect on those outside it. Even worse, is that there are some stories in which inflicting pain within a simulation is rewarded outside of it. Here’s a possible hypothetical:
Imagine humans in base reality create friendly AI. To respect their past, the humans ask the AI to create tons of sims living in different eras. Since some historical info was lost to history, the sims are slightly different from base reality. Therefore, in each sim, there’s a chance AI never becomes aligned. Accounting for this possibility, base reality humans decide to end sims in which AI becomes misaligned and replace those sims with paradise sims where everyone is happy.
In the above scenario, both total and average utilitarianism would recommend intentionally creating misaligned AI so that paradise ensues.
I’m sure you can craft even more plausible stories.
My point is, even if our understanding of physics and logic is correct, I don’t see why we ought to privilege the hypothesis that simulations are memories. I also don’t see why we ought to privilege the idea that it’s in our interest to increase utility within the simulation. Can you please clarify why you’re so confident about these notions?
Thank you
I have to say, quila, I’m pleasantly surprised that your response above is both plausible and logically coherent—qualities I couldn’t find in any of the Reddit responses. Thank you.
However, I have concerns and questions for you.
Most importantly, I worry that if we’re currently in a simulation, physics and even logic could be entirely different from what they appear to be. If all our senses are illusory, why should our false map align with the territory outside the simulation? A story like your “Mutual Anthropic Capture” offers hope: a logically sound hypothesis in which our understanding of physics is true. But why should it be? Believing that a simulation exactly matches reality sounds to me like the privileging the hypothesis fallacy.
By the way, I’m also somewhat skeptical of a couple of your assumptions in Mutual Anthropic Capture. Still, I think it’s a good idea overall, and some subtle modifications to the idea would probably make logically sound. I won’t bother you about those small issues here, though; I’m more interested in your response to my concern above.
This is an interesting observation which may well be true, I’m not sure, but the more intuitive difference is that SSA is about actually existing observers, while SIA is about potentially existing observers. In other words, if you are reasoning about possible realities in the so-called “multiverse of possibilities,” than you are using SIA. Whereas if you are only considering a single reality (e.g., the non-simulated world), you select a reference class from that reality (e.g., humans), you may choose to use use SSA to say that you are a random observer from that class (e.g., a random human in human history).
You are describing the SIA assumption to a T.
This is what I was thinking:
If simulations exist, we are choosing between two potentially existing scenarios, either I’m the only real person in my simulation, or there are other real people in my simulation. Your argument prioritizes the latter scenario because it contains more observers, but these are potentially existing observers, not actual observers. SIA is for potentially existing observers.
I have a kind of intuition that something like my argument above is right, but tell me if that is unclear.
And note: one potential problem with your reasoning is that if we take it to it’s logical extreme, it would be 100% certain that we are living in a simulation with infinite invisible observers. Because infinity dominates all the finite possibilities.
I think you are overlooking that your explanation requires BOTH SSA and SIA, but yes, I understand where you are coming from.
Other people here have responded in similar ways to you; but the problem with your argument is that my original argument could also just consider only simulations in which I am the only observer. In which case Pr(I’m distinct | I’m in a simulation)=1, not 0.5. And since there’s obviously some prior probability of this simulation being true, my argument still follows.
I now think my actual error is saying Pr(I’m distinct | I’m not in a simulation)=0.0001, when in reality this probability should be 1, since I am not a random sample of all humans (i.e., SSA is wrong), I am me. Is that clear?
Lastly, your final paragraph is akin to the SSA + SIA response to the doomsday paradox, which I don’t think is widely accepted since both those assumptions lead to a bunch of paradoxes.
True, but that wasn’t my prior. My assumption was that if I’m in a simulation, there’s quite a high likelihood that I would be made to be so ‘lucky’ to be the highest on this specific dimension. Like a video game in which the only character has the most Hp.
I have thought a lot about anthropics.
In an infinite universe, there are an infinite number of identical observers. You cannot define a probability distribution with an infinite sample space, and infinite cardinalities cannot help you. You cannot ask for the probability that it is Monday or Tuesday upon flipping tails because there are an infinite number of observers in both cases.
Do you agree that anthropic questions like these are meaningless if we live in an infinite universe?