My current best-understanding is that if we assume people have arbitrary inconsistencies, it will be impossible to do better than satisfice on different human values by creating near-pareto improvements for intra-human values. But inconsistent values don’t even allow pareto-improvements! Any change makes things incomparable. Given that, I think we do need a super-prospect theory that explains in a systematic way what humans do “wrong” so that we can pick what an AI should respect of human preferences, and what can be ignored.
For instance, I love my children, and I like chocolate. I’m also inconsistent with my preferences in ways that differs; at a given moment of time, I’m much more likely to be upset with my kids and not want them around than I am to not want chocolate. I want the AI to respect my greater but inconsistent preference for my children over the more consistent preference for candy. I don’t know how to formalize this in a way that would generalize, which seems like a problem. Similar problems exist for time preference and similar typical inconsistencies—they are either inconsistent, or at least can be exploited by an AI that has a model which doesn’t think about resolving those inconsistencies.
With a super-prospect theory, I would hope we may be able to define a CEV or similar, which allows large improvements by ignoring the fact that those improvements are bad for some tiny part of my preferences. And perhaps the AI should find the needed super-prospect theory and CEV—but I am deeply unsure about the safety of doing this, or the plausibility of trying to solve it first.
(Beyond this, I think we need to expect that between-human values will differ, and we can keep things safe by insisting on a near-pareto improvement, only things that are a pareto improvement with respect to a very large portion of people, and relatively minor dis-improvements for the remainder. But that’s a different discussion.)
My current best-understanding is that if we assume people have arbitrary inconsistencies, it will be impossible to do better than satisfice on different human values by creating near-pareto improvements for intra-human values. But inconsistent values don’t even allow pareto-improvements! Any change makes things incomparable. Given that, I think we do need a super-prospect theory that explains in a systematic way what humans do “wrong” so that we can pick what an AI should respect of human preferences, and what can be ignored.
For instance, I love my children, and I like chocolate. I’m also inconsistent with my preferences in ways that differs; at a given moment of time, I’m much more likely to be upset with my kids and not want them around than I am to not want chocolate. I want the AI to respect my greater but inconsistent preference for my children over the more consistent preference for candy. I don’t know how to formalize this in a way that would generalize, which seems like a problem. Similar problems exist for time preference and similar typical inconsistencies—they are either inconsistent, or at least can be exploited by an AI that has a model which doesn’t think about resolving those inconsistencies.
With a super-prospect theory, I would hope we may be able to define a CEV or similar, which allows large improvements by ignoring the fact that those improvements are bad for some tiny part of my preferences. And perhaps the AI should find the needed super-prospect theory and CEV—but I am deeply unsure about the safety of doing this, or the plausibility of trying to solve it first.
(Beyond this, I think we need to expect that between-human values will differ, and we can keep things safe by insisting on a near-pareto improvement, only things that are a pareto improvement with respect to a very large portion of people, and relatively minor dis-improvements for the remainder. But that’s a different discussion.)