Yes, I agree with Ryan here. If you want to make good decisions to improve the world in an impartial way, then in your thought experiment, you should act like the operation happened and you were the real Chosen One.
If you want to be selfish in an indexical way, then things get confusing, but I think selfishness with indexical preferences is not a very coherent and reasonable concept.
The thing I’m trying to explain is when and when not you should act surprised and re-evaluate your world-model. If Bob thinks of himself as being sampled from conscious continuations, he will waste time acting surprised if he wakes up with the Sword. I claim it’s better to calibrate your surprisal to as if you were being sampled weighted by importance—then, he won’t waste crucial time if he wakes up with the Sword. This is a pretty basic case of optimizing for expected values, but people often don’t think of surprisal in these terms, and they get confused about whether they should be surprised they are not a fish, and sometimes they even draw wrong conclusions like “fish must not be conscious, otherwise it would be overwhelmingly likely I am a fish”. I’m trying to illustrate why you shouldn’t act surprised that you are important.
I added “when you think about how surprised you should feel about something” to the sentence, which hopefully makes it clearer.
An intuition pump on anthropics.
In some recent conversations with friends, I was asked some questions of the type: “If most conscious beings on Earth are fish, why am I not a fish? If we expect gazillion digital minds to live in the future, why am I not a digital mind in the future? Isn’t it very surprising that we seem to live close to the hinge of history?”
My position is that when you consider how surprised you should feel about something, you shouldn’t think of your current experience moment as being sampled from the set of all conscious experience moments. Instead, you should think of your current decision as being sampled from the set of all decisions, weighted by their importance.
I wrote up my more detailed thoughts on these questions in my recent sequence, but I wanted to give a shorter, evocative argument here.
—
Imagine a friend, Bob, knowingly undergoing an operation where a billion copies of him are created, all sharing his previous memories and personality. 999,999,999 copies will wake up as powerless farmers on isolated islands, not being able to affect anything in the outside world. One copy will wake up with the Sword of the Chosen One in his hand, and the fate of the world will depend on his actions.
You sit down with Bob before he undergoes the operation. You both agree that the actions of the farmer-copies are not very important, so you focus on making plans for the Chosen One, plans that the version of him waking up with a Sword in his hand should execute without delay.
You are watching the operation from the outside, not being able to interact with Bob anymore. The operation goes exactly as planned. The billion copies all wake up.
Now imagine the one copy with the Sword waking up and instead of following the agreed-upon plan, wallowing in doubt. “Why am I the Chosen One and not a farmer?” he asks. He looks for explanations for this strange coincidence. He is thinking through hare-brained possibilities where the operation didn’t go as planned, so fewer farmers were created, which would make it less surprising that he is not a farmer.
“You fool!” you would want to yell at Bob. “We had a plan. The operation went as expected. Looking from the outside, it’s still as obvious as ever what the Chosen One needs to do. You gained no new information compared to me looking from the outside. What are you thinking about then? Don’t waste your time; do your duty.”
—
I believe we are in a very similar situation to Bob. I believe there is no explanation to the question why you were not born a fish or a digital mind in the future, just like there is no explanation why Bob With the Sword did not wake up as a farmer. It was always knowable that some beings will make important decisions, living on the hinge of history, while others beings will be, well, fish. It is, by definition, more important that the more important decisions get made well. So from an impartial perspective, you should think as if your current decision was sampled from all decisions weighted by importance.
Generally, if you find yourself in a position that should be very rare according to your world model, that’s a good moment to feel surprise, re-examine your world model and look for an explanation. But finding yourself in a position of unusually high influence is a special case that requires no explanation—it is not surprising if you imagine your decision being sampled weighted by importance.
It looks like you are one of the rare beings (not a fish, and not a digital mind in the stable future) who has a chance of making a noticeable difference to the fate of the world. Don’t waste your time; do your duty.
I think 2.6 billion people in the US would probably do fine. There is just a lot of land to be farmed in the US. It’s not even clear to me if per capita wealth would be lower or higher if the US had 2.6 billion people—my guess is higher. So I think the miracle leading to a wealthier society today really was economic and technological, and not contraception.
Ok, in my eventual formulation, I’m not actually relying on moral parliaments. However, I will note that in your and Paul’s original formulation of the double-updating paradox, you were writing about bets where your utility is linear in money (e.g. small amounts of money donated to charity).
Here you write “if you ever get evidence that a decision is high-stakes enough and universal enough throughout the universe, EDT may well recommend that you make an exception for it and do a proper double-update on it, which seems bad”. At the point where you are making very high-stakes and universally applicable decisions, I’m no longer convinced you get linearity, and I might bite the bullet that there might be some bets you don’t want to engage in on either side, or the paradox fails in other ways.
Altogether, I still think that my previous answer basically dissolves this paradox.
I think in both cases the decision-correlational reference class you should take into account is not just you learning T1 is true and you learning T2 is true within this particular experiment. It’s every instance across the multiverse where beings similar to you need to make bets about questions they have no clue about. Taking all these correlations into account, the correct thing to do is to bet with 50-50.
Thanks! The statue example seems right, in that case the shortest description really doesn’t point to something I would call an observer that can be called me, since it points to the inanimate inscription on the statue.
Anyway, as you can see from my later posts, I’m generally not in favor of following Solomonoff induction off the cliff, and I think we will probably need to use more qualitative value judgments in our weightings.
I don’t think AI water usage fits the pattern. People keep repeating the line “AI uses a lot of water”, and making political cartoons about AI slurping up all the water in the world, but I don’t remember anyone ever arguing for it being true.
This is in contrast to religion, price fixing and possibility of AGI, where you can in fact find smart people making long and complicated arguments on both sides.
Thanks! Here is the third and final post.
To be clear, I’m not really endorsing this logic, and I will write more in my next post about how to compare the values of smaller and bigger universes. But I do think that naive expected value calculation leads to weird places with infinities, so this is a good motivating question for developing better theories.
I guess this is largely a semantic question at this point. Originally, I wrote “the laws of nature”, and not “our laws of nature”. But even if you say “our world”, arguably that still works: if I’m living on a computer run by aliens, then arguably the base reality where the computer sits is my world. You can point to me by pointing at the laws of the base reality world and pointing at my computer within it.
I’m interested though why you believe there are shorter good predictors than world + pointer. I agree it’s possible, I just can’t think of one. How would they look like?
Thank you! I edited the post with the link.
Yes, I absolutely think that Solomonoff induction is probably malign. But I understood that to also work through a world + pointer framework. There is someone in our world making a very important decision about the future, based on a prediction they are making using Solomonoff induction. The pointer to their moment is not particularly simple. Alines in another world run a short solipsistic simulation of this entity making the decision. They run the simulation in a particularly easy-to-point-to sport, or they run many copies of the simulation in random places. Thereby the aliens simulation-capture the predictor.
https://www.lesswrong.com/posts/KSdqxrrEootGSpKKE/the-solomonoff-prior-is-malign-is-a-special-case-of-a
Is your understanding different?
Great point, I haven’t thought of this. I will edit the post, referencing you.
I do think that the constant helps a lot: we now moved from “you are overwhelmingly likely to be a BB” to “BB version of you have 2^-20 weight, so are basically negligible given that you can’t do much for them anyway”.
I still think that there is something unreasonable about giving 2^-20 weight to BBs based on this logic—I think the weirdness is pretty similar to the World-summoning by writing thought experiment. I think I just want to reject this type of logic by fiat under my non-realist epistemology that I write about at the end of the post.
Maybe I’m missing something, but yes, I think “the laws of nature plus a pointer to my location” is a good description. Why would that be so bizarre? It doesn’t seem crazy to define the laws of nature, then define the concept of temperature, the concept of distance and the concept of time, then define the coldest spot in the universe’s history (which is, iirc, to the best of our knowledge, human-created), then specify a spot X years and Y kilometers distance from that experiment, and that points out me sitting here right now. (Assuming that we never create something colder in the future.) I think something like that is what this description of Solomonoff induction would do?
I agree that the quantum multiverse complicates things, I am writing about that in my next post.
---
In my head, there are two definitions of Solomonoff induction: one is where you assume that your inputs so far has been produced by a simple program. My impression is that that’s the definition I usually encountered in the Bay Area folklore, and for example I think that’s the definition Joe Carlsmith is using in his post on UDASSA. So that’s the version I reference in this post.
The other definition, where you assume you are sampled from an easy-to-describe random distribution was something I don’t remember encountering, and came up with it as an exciting new thing. I write about it in my next post. https://www.lesswrong.com/posts/zvqodicK8q2SNAZkd/infinite-ethics-and-udassa#Resolution___Solomonoff_over_distributions
I think that’s probably equivalent to what you describe as “the UTM version of Solomonoff”. As I say in that post, it looks like this is maybe the more standard version in academia, and in retrospect maybe I should have started with that version in the first post.
But I don’t think the distinction between the two versions are a crux for any of my objections, except for objection 3, which I already flag in the first post that it will be relatively easy to resolve with a new formalism.
(It’s possible I’m still missing something here—as you might be able to tell, I don’t have that much background in the mathematical study of Solomonoff-induction.)
I like my strategy of silently amending “assuming a materialistic world-view and not being in a sim” to all mundwne questions. I think that makes it possible to non-misleadingly communicate about the probability of AI takeover.
(This is assuming that you agree that you should mostly act as if you were in a materialistic non-sim world.)
Yes, I will make this point in my next post. (I’m not sure though if probabilities being caring measures is a necessary consequence of UDT. I thought this was a different axis.)
Ok, but are there specific chapters or pages he or others could link to? Linking to long novels is not very useful.
Can you share a brief description or a link to your favored metaphysics of reality-fluid? I haven’t yet come across good write-ups on this topic.
I currently think I don’t actually need to explain what I see. What is the advantage of explaing it? Under normal circumstances, it’s useful to explain why I see e.g. the washing machine not working, because that gives me useful predictions about how the washing machine will behave in the future, and knowing that is useful for fulfiling my values.
But if I have a preference like “life only has meaning if I win the lottery”, then I think it’s not action-relevant to find an explanation if I see myself winning. The day before the lottery, I was already only making follow-up plans for what to do if I win—that’s the only case where life has meaning. And it was not part of the plan to do an investigation of why I won the lottery—why would I want to waste my time on that? So when I actually win, I in fact don’t investigate it further.
To be clear, I think that caring a lot about universes where you win the lottery, at the expense of other universes, is a very sillly moral belief. If someone held this belief, I would tell them to meditate a little and play with a toddler and imagine if they would really care less about her if she was living in a universe where they don’t win the lottery. But if they come back saying that they genuinely only care about lottery-winning universes, then I can’t argue with their utility function.
I think having at least 0.1% of my caring allocated to short description-length space-time points is not that crazy of a moral belief. And once I have that, it’s reasonable to have a policy like “if I see an ordered world, I will act as if I was not a Boltzmann-brain, because these ordered situations are important for the 0.1% of my values, but are vanishingly rare and unimportant from the perspective of the 99.9%”.
I now see an ordered world, so in accordance with this policy, I act as if I was not a Boltzmann-brain. Acting as if I was not a Boltzmann-brain includes not looking for mechanistic explanations for why I’m not one, so I don’t do that.
I don’t know, maybe I phrased things wrongly. I’m in favor trying to do the all-things-condidered-best I can, and optimizing for expected values. And of course, you shouldn’t do things lime imagining you are sampled by importance-weighting, and then calcilate expected values: that would be double-updating.
All I am trying to say is that this is a useful perspeczive to take when deciding what should make you surprised and make you reevaluate your world model.