Yep, sorry, I saw −3, −2, −1, etc… and concluded you weren’t doing the 2 jumps; my bad!
Then somehow the work is just postponed to the point where we try to combine partial preferences?
Yes. But unless we have other partial preferences or meta-preferences, then the only resonable way of combining them is just to add them, after weighting.
I like your reciprocal weighting formula. It seems to have good properties.
“How many copies of people like me are there in each universe?”
Then as long as your copies know that 3K has been observed, and excluding simulations and such, the answers are “(a lot, a lot, not many, not many)” in the four universes (I’m interpreting “die off before spreading through space” as “die off just before spreading through space”).
This is the SIA answer, since I asked the SIA question.
Each partial preference is meant to represent a single mental model inside the human, with all preferences weighted the same (so there can’t be “extremely weak” preferences, compared with other preference in the same partial preference). Things like “increased income is better”, “more people smiling is better”, “being embarrassed on stage is the worse”.
We can imagine a partial preference with more internal structure, maybe internal weights, but I’d simply see that as two separate partial preferences. So we’d have the utilities you gave to through to for one partial preference (actually, my formula doubles the numbers you gave), and , , for the other partial preference—which has a very low weight by assumption. So the order of and is not affected.
EDIT: I’m pretty sure we can generalise my method for different weights of preferences, by changing the formula that sums the squares of utility difference.
Neat!
Though I should mention that my current version of partial preferences does not assume all cycles are closed—the constrained optimisation can be seen as trying to get “as close as possible” to that, given non-closed cycles.
What’s the set of answers, and how are they assessed?
I encourage you to submit other ideas anyway, since your ideas are good.
Not sure yet about how all these things relate; will maybe think of that more later.
Hey there! Thanks for your long comment—but, alas, this model of partial preferences is obsolete :-(
Because of other problems with this, I’ve replaced it with the much more general concept of a preorder. This can express all the things we want to express, but is a lot less intuitive for how humans model things. I may come up with some alternative definition at some point (less general than a preorder, but more general than this post.
Thanks for the comment in any case.
I find it hard to imagine that you’re actually denying that you or I have things that, colloquially, one would describe as preferences, and exist in an objective sense.
I deny that a generic outside observer would describe us as having any specific set of preferences, in an objective sense.
This doesn’t bother me too much, because it’s sufficient that we have preferences in a subjective sense—that we can use our own empathy modules and self-reflection to define, to some extent, our preferences.
a brain is ultimately many fewer assumptions (to the pre-industrial Norse people)
“Realistic” preferences make ultimately fewer assumptions (to actual humans) that “fully rational” or other preference sets.
The problem is that this is not true for generic agents, or AIs. We have to get the human empathy module into the AI first—not so it can predict us (it can already do that through other means), but so that its decomposition of our preferences is the same as ours.
Can you make this a bit more general, rather than just for the specific example?
For low bandwidth, you have to specify the set of answers that are available (and how they would be checked).
I tried to answer in more detail here: https://www.lesswrong.com/posts/f5p7AiDkpkqCyBnBL/preferences-as-an-instinctive-stance (hope you didn’t mind; I used your comment as a starting point for a major point I wanted to clarify).
But I admit to being confused now, and not understanding what you mean. Preferences don’t exist in the territory, so I’m not following you, sorry! :-(
Saying that an agent has a preference/reward R is an interpretation of that agent (similar to the “intentional stance” of seeing it as an agent, rather than a collection of atoms). And the (p,R) and (-p,-R) interpretations are (almost) equally complex.
I disagree. I think that if we put a complexity upper bound on human rationality, and assume noisy rationality, then we will get values that are “meaningless” from your perspective.
I’m trying to think of ways how of we could test this....
Imagine [...] distinguish.
It’s because of concerns like this that we have to solve the symbol grounding problem for the human we are trying to model; see, eg, https://www.lesswrong.com/posts/EEPdbtvW8ei9Yi2e8/bridging-syntax-and-semantics-empirically
But that doesn’t detract from the main point: that simplicity, on its own, is not sufficient to resolve the issue.
If you assume that human values are simple (low komelgorov complexity) and that human behavior is quite good at fulfilling those values, then you can deduce non trivial values for humans.
And you will deduce them wrong. “Human values are simple” pushes you towards “humans have no preferences”, and if by “human behavior is quite good at fulfilling those values” you mean something like noisy rationality, then it will go very wrong, see for example https://www.lesswrong.com/posts/DuPjCTeW9oRZzi27M/bounded-rationality-abounds-in-models-not-explicitly-defined
And if instead you mean a proper accounting of bounded rationality, of the difference between anchoring bias and taste, of the difference between system 1 and system 2, of the whole collection of human biases… well, then, yes, I might agree with you. But that’s because you’ve already put all the hard work in.
I’ve thinking of a rather naive form of preference utilitarianism, of the sort “if the human agree to it or choose it, then it’s ok”. In particular, you can end up with some forms of depression where the human is miserable, but isn’t willing to change.
I’ll clarify that in the post.
I’m finding these “is the correct utility function” hard to parse. Humans have a bit of and a bit of . But we are underdefined systems; there is no specific value of that is “true”. We can only assess the quality of using other aspects of human underdefined preferences.
This seems way too handwavy.
It is. Here’s an attempt at a more formal definition: humans have collections of underdefined and somewhat contradictory preferences (using preferences in a more general sense than preference utilitarianism). These preferences seem to be stronger in the negative sense than in the positive: humans seem to find the loss of a preference much worse than the gain. And the negative is much more salient, and often much more clearly defined, than that positive.
Given that maximising one preference tends to put the values of others at extreme values, human overall preferences seem better captured by a weighted mix of preferences (or a smooth min of preferences) than by any single preference, or small set of preferences. So it is not a good idea to be too close to the extremes (extremes being places where some preferences have weight put on them).
Now there may be some sense in which these extreme preferences are “correct”, according to some formal system. But this formal system must reject the actual preferences of humans today; so I don’t see why these preferences should be followed at all, even if they are correct.
Ok, so the extremes are out; how about being very close to the extremes? Here is where it gets wishywashy. We don’t have a full theory of human preferences. But, according to the picture I’ve sketched above, the important thing is that each preference gets some positive traction in our future. So, yes to might no mean much (and smooth min might be better anyway). But I believe I could say:
There are many weighted combinations of human preferences that are compatible with the picture I’ve sketched here. Very different outcomes, from the numerical perspective of the different preferences, but all falling within an “acceptability” range.
Still a bit too handwavy. I’ll try and improve it again.
If we set aside infinity, which I don’t know how to deal with, then the SIA answer does not depend on utility bounds—unlike my anthropic decision theory post.
Q1: “How many copies of people (currently) like me are there in each universe?” is well-defined in all finite settings, even huge ones.
No, I mean not many, as compared with how many there are in universes 1 and 2. Other observers are not relevant to Q1.
I’ll reiterate my claim that different anthropic probability theories are “correct answers to different questions”: https://www.lesswrong.com/posts/nxRjC93AmsFkfDYQj/anthropic-probabilities-answering-different-questions