Of course, this is just a heuristic argument, and if partial preference orderings in real life have some special structure, the conclusion might differ.
Hmm I may be missing something here, but I suspect that “partial preference orderings in real life have some special structure” in the relevant sense, is very likely true. Human preferences don’t appear to be a random sample from the set of all possible partial orders over “world states” (or more accurately, human models of worlds).
First of all, if you model human preferences as a vector-valued utility function (i.e. one element of the vector per subagent) it seems that it has to be continuous, and probably Lipschitz, in the sense that we’re limited in how much we can care about small changes in the world state. There’s probably some translation of this property into graph theory that I’m not aware of.
Also, it seems like there’s one or a handful of preferred factorizations of our world model into axes-of-value, and different subagents will care about different factors/axes. More specifically, it appears that human preferences have a strong tendency to track the same abstractions that we use for empirical prediction of the world; as John says, human values are a function of humans’ latent variables. If you stop believing that souls and afterlives exist as a matter of science, it’s hard to continue sincerely caring about what happens to your soul after you die. We also don’t tend to care about weird contrived properties with no explanatory/predictive power like “grue” (green before 1 January 2030 and blue afterward).
To the extent this is the case, it should dramatically– exponentially, I think– reduce the number of posets that are really possible and therefore the number of subagents needed to describe them.
Hmm I may be missing something here, but I suspect that “partial preference orderings in real life have some special structure” in the relevant sense, is very likely true. Human preferences don’t appear to be a random sample from the set of all possible partial orders over “world states” (or more accurately, human models of worlds).
First of all, if you model human preferences as a vector-valued utility function (i.e. one element of the vector per subagent) it seems that it has to be continuous, and probably Lipschitz, in the sense that we’re limited in how much we can care about small changes in the world state. There’s probably some translation of this property into graph theory that I’m not aware of.
Also, it seems like there’s one or a handful of preferred factorizations of our world model into axes-of-value, and different subagents will care about different factors/axes. More specifically, it appears that human preferences have a strong tendency to track the same abstractions that we use for empirical prediction of the world; as John says, human values are a function of humans’ latent variables. If you stop believing that souls and afterlives exist as a matter of science, it’s hard to continue sincerely caring about what happens to your soul after you die. We also don’t tend to care about weird contrived properties with no explanatory/predictive power like “grue” (green before 1 January 2030 and blue afterward).
To the extent this is the case, it should dramatically– exponentially, I think– reduce the number of posets that are really possible and therefore the number of subagents needed to describe them.