I’ll address your points in reverse order.
What if all of the worlds with the lowest EU are completely bizarre (like, boltzmann brains, or worlds that have somehow fallen under the rule of fantastical devils with literally no supporters).
The Boltzmann brain issue is addressed in infra-Bayesian physicalism with a “fairness” condition that excludes worlds from the EU calculation where you are run with fake memories or the history of your actions is inconsistent with what your policy says you would actually do. Vanessa talks about this in AXRP episode 14. The “worlds that have somehow fallen under the rule of fantastical devils” thing is only a problem if that world is actually assigned high measure in one of the sa-measures (fancy affine-transformed probability distributions) in your prior. The maximin rule is only used to select the sa-measure in your convex set with lowest EU, and then you maximize EU given that distribution. You don’t pick the literal worst conceivable world.
Notably, if you don’t like the maximin rule, it’s been shown in Section 4 of this post that infra-Bayesian logic still works with optimism in the face of Knightian uncertainty, it’s just that you don’t get worst-case guarantees anymore. I’d suspect that you could also get away with something like “maximize 10th percentile EU” to get more tempered risk-averse behavior.
Solomonoff inducting, producing an estimate of the measure of my existence (the rate of the occurrence of the experience I’m currently having) across all possible universe-generators weighted inversely to their complexity seems totally coherent to me. (It’s about 0.1^10^10^10^10)
I’m not sure I follow your argument. I thought your view was that minds implemented in more places, perhaps with more matter/energy, have more anthropic measure? The Kolmogorov complexity of the mind seems like an orthogonal issue.
Maybe you’re already familiar with it, but I think Stuart Armstrong’s Anthropic Decision Theory paper (along with some of his LW posts on anthropics) do a good job of “deflating” anthropic probabilities and shifting the focus to your values and decision theory.
Hmm I may be missing something here, but I suspect that “partial preference orderings in real life have some special structure” in the relevant sense, is very likely true. Human preferences don’t appear to be a random sample from the set of all possible partial orders over “world states” (or more accurately, human models of worlds).
First of all, if you model human preferences as a vector-valued utility function (i.e. one element of the vector per subagent) it seems that it has to be continuous, and probably Lipschitz, in the sense that we’re limited in how much we can care about small changes in the world state. There’s probably some translation of this property into graph theory that I’m not aware of.
Also, it seems like there’s one or a handful of preferred factorizations of our world model into axes-of-value, and different subagents will care about different factors/axes. More specifically, it appears that human preferences have a strong tendency to track the same abstractions that we use for empirical prediction of the world; as John says, human values are a function of humans’ latent variables. If you stop believing that souls and afterlives exist as a matter of science, it’s hard to continue sincerely caring about what happens to your soul after you die. We also don’t tend to care about weird contrived properties with no explanatory/predictive power like “grue” (green before 1 January 2030 and blue afterward).
To the extent this is the case, it should dramatically– exponentially, I think– reduce the number of posets that are really possible and therefore the number of subagents needed to describe them.