Start off with CEV-0. I won’t go into how that is generated, but it will have a lot of arbitrary decisions and stuff that seems vaguely sensible.
Then ask CEV-0 the following questions:
How should CEV-1 go about aggregating people’s preferences?
How should CEV-1 deal with non-transitive or non-independent preferences?
How should CEV-1 determine preferences between outcomes that the subject could never have imagined?
What should CEV-1 do if people lack the expertise to judge the long-term consequences of their preferences?
Should CEV-1 consider people’s stated or revealed preferences, or both?
Should CEV-1 consider preferences of non-human animals, people in comas, etc.?
How should CEV-1 deal with people who seem to be trying to modify their own preferences in order to game the system? (utility monsters/tactical voting)
… and so on. The answers to these questions then make up CEV-1. And then CEV-1 is asked the same questions to produce CEV-2.
Various different things could happen here. It could converge to a single stable fixed point. It could oscillate. It could explode, diverging wildly from anything we’d consider reasonable. Or its behavior could depend on the initial choice of CEV-0 (e.g. multiple attractive fixed points).
Explosion could (possibly) be avoided by requiring CEV-n to pass some basic sanity checks, though that has problems too (the sanity checks may not be valid, i.e. they just reflect our own biases. Or they may not be enough—they act as constraints on the evolution of the system but it could still end up insane in respects we haven’t anticipated).
Some other problems could be resolved by asking CEV-n how to resolve them.
I’m not sure how to deal with the multiple stable fixed points case. That would seem to correspond to different cultures or special interest groups all trying to push whichever meta-level worldview benefits them the most.
What about recursive CEV?
Start off with CEV-0. I won’t go into how that is generated, but it will have a lot of arbitrary decisions and stuff that seems vaguely sensible.
Then ask CEV-0 the following questions:
How should CEV-1 go about aggregating people’s preferences?
How should CEV-1 deal with non-transitive or non-independent preferences?
How should CEV-1 determine preferences between outcomes that the subject could never have imagined?
What should CEV-1 do if people lack the expertise to judge the long-term consequences of their preferences?
Should CEV-1 consider people’s stated or revealed preferences, or both?
Should CEV-1 consider preferences of non-human animals, people in comas, etc.?
How should CEV-1 deal with people who seem to be trying to modify their own preferences in order to game the system? (utility monsters/tactical voting)
… and so on. The answers to these questions then make up CEV-1. And then CEV-1 is asked the same questions to produce CEV-2.
Various different things could happen here. It could converge to a single stable fixed point. It could oscillate. It could explode, diverging wildly from anything we’d consider reasonable. Or its behavior could depend on the initial choice of CEV-0 (e.g. multiple attractive fixed points).
Explosion could (possibly) be avoided by requiring CEV-n to pass some basic sanity checks, though that has problems too (the sanity checks may not be valid, i.e. they just reflect our own biases. Or they may not be enough—they act as constraints on the evolution of the system but it could still end up insane in respects we haven’t anticipated).
Some other problems could be resolved by asking CEV-n how to resolve them.
I’m not sure how to deal with the multiple stable fixed points case. That would seem to correspond to different cultures or special interest groups all trying to push whichever meta-level worldview benefits them the most.
Once CEV-n becomes a utility function, it will generally (but not always) get stuck there for ever.