So let us specify a probability distribution over the space of all possible desires. If we accept the orthogonality thesis, we should not want this probability distribution to build in any bias towards certain kinds of desires over others. So let’s spread our probabilities in such a way that we meet the following three conditions. Firstly, we don’t expect Sia’s desires to be better satisfied in any one world than they are in any other world. Formally, our expectation of the degree to which Sia’s desires are satisfied at W is equal to our expectation of the degree to which Sia’s desires are satisfied at W∗, for any W,W∗. Call that common expected value μ″. Secondly, our probabilities are symmetric around μ. That is, our probability that W satisfies Sia’s desires to at least degree μ+x is equal to our probability that it satisfies her desires to at most degree μ−x. And thirdly, learning how well satisfied Sia’s desires are at some worlds won’t tell us how well satisfied her desires are at other worlds. That is, the degree to which her desires are satisfied at some worlds is independent of how well satisfied they are at any other worlds. (See the appendix for a more careful formulation of these assumptions.) If our probability distribution satisfies these constraints, then I’ll say that Sia’s desires are ‘sampled randomly’ from the space of all possible desires.
This is a characterization, and it remains to show that there exist distributions that fit it (I suspect there are not, assuming the sets of possible desires and worlds are unbounded).
I also find the 3rd criteria counterintuitive. If worlds share features, I would expect these to not be independent.
I also find the 3rd criteria counterintuitive. If worlds share features, I would expect these to not be independent.
Agreed, I think this criterion is very strongly violated in practice; a Gaussian process prior with a nontrivial covariance function would be a bit more realistic.
This is a characterization, and it remains to show that there exist distributions that fit it (I suspect there are not, assuming the sets of possible desires and worlds are unbounded).
I also find the 3rd criteria counterintuitive. If worlds share features, I would expect these to not be independent.
Agreed, I think this criterion is very strongly violated in practice; a Gaussian process prior with a nontrivial covariance function would be a bit more realistic.