Basically agree with Paul, and I especially want to note that I’ve barely thought about it and so this would likely change a ton with more information. To put some numbers of my own:
“No generalization”: 65%
“Some generalization”: 5% (I don’t actually have stories where this is an outcome; this is more like model uncertainty)
“Lots of generalization, but something went wrong”: 30%
These are from my own perspective of what these categories mean, which I expect are pretty different from yours—e.g. maybe I’m at ~2% that upon reflection I’d decide that hedonium is great and so that’s actually perfect generalization; in the last category I include lots of worlds that I wouldn’t describe as “existing humans recognizably survive”, e.g. we decide to become digital uploads, then get lots of cognitive enhancements, throw away a bunch of evolutionary baggage, but also we never expand to the stars because AI has taken control of it and given us only Earth.
I think the biggest avenues for improving the answers would be to reflect more on the kindness + cooperation and acausal trade stories Paul mentions, as well as the possibility that a few AIs end up generalizing close to correctly and working out a deal with other AIs that involves humanity getting, say, Earth.
Given that not every biological civilization solves the problem, what does the rest of the multiverse look like?
If we’re imagining civilizations very similar to humanity, then the multiverse looks like ~100% of one of the options. Reality’s true answer will be very overdetermined; it is a failure of our map that we cannot determine the answer. I don’t know much about quantum physics / many-worlds, but I’d be pretty surprised if small fluctuations to our world made a huge difference; you’ll need a lot of fluctuations adding up to a lot of improbability before you affect a macro-level property like this, unless you just happen to already be on the knife-edge.
If the biological civilizations could be very different from ours, then I have no idea how to quickly reason about this question and don’t have an answer, sorry.
If we’re imagining civilizations very similar to humanity, then the multiverse looks like ~100% of one of the options. Reality’s true answer will be very overdetermined; it is a failure of our map that we cannot determine the answer. I don’t know much about quantum physics / many-worlds, but I’d be pretty surprised if small fluctuations to our world made a huge difference; you’ll need a lot of fluctuations adding up to a lot of improbability before you affect a macro-level property like this, unless you just happen to already be on the knife-edge.
This doesn’t contradict anything you’re saying but there’s arguably a wager for thinking that we’re on the knife-edge – our actions are more impactful if we are.
[Edit to add point:] The degree to which any particular training approach generalizes is of course likely a fixed fact (like in the Lesswrong post you link to about fire). But different civilizations could try different training approaches, which produces heterogeneity for the multiverse.
Basically agree with Paul, and I especially want to note that I’ve barely thought about it and so this would likely change a ton with more information. To put some numbers of my own:
“No generalization”: 65%
“Some generalization”: 5% (I don’t actually have stories where this is an outcome; this is more like model uncertainty)
“Lots of generalization, but something went wrong”: 30%
These are from my own perspective of what these categories mean, which I expect are pretty different from yours—e.g. maybe I’m at ~2% that upon reflection I’d decide that hedonium is great and so that’s actually perfect generalization; in the last category I include lots of worlds that I wouldn’t describe as “existing humans recognizably survive”, e.g. we decide to become digital uploads, then get lots of cognitive enhancements, throw away a bunch of evolutionary baggage, but also we never expand to the stars because AI has taken control of it and given us only Earth.
I think the biggest avenues for improving the answers would be to reflect more on the kindness + cooperation and acausal trade stories Paul mentions, as well as the possibility that a few AIs end up generalizing close to correctly and working out a deal with other AIs that involves humanity getting, say, Earth.
If we’re imagining civilizations very similar to humanity, then the multiverse looks like ~100% of one of the options. Reality’s true answer will be very overdetermined; it is a failure of our map that we cannot determine the answer. I don’t know much about quantum physics / many-worlds, but I’d be pretty surprised if small fluctuations to our world made a huge difference; you’ll need a lot of fluctuations adding up to a lot of improbability before you affect a macro-level property like this, unless you just happen to already be on the knife-edge.
If the biological civilizations could be very different from ours, then I have no idea how to quickly reason about this question and don’t have an answer, sorry.
This doesn’t contradict anything you’re saying but there’s arguably a wager for thinking that we’re on the knife-edge – our actions are more impactful if we are.
[Edit to add point:] The degree to which any particular training approach generalizes is of course likely a fixed fact (like in the Lesswrong post you link to about fire). But different civilizations could try different training approaches, which produces heterogeneity for the multiverse.