At the FHI we’re looking for sensible ways of normalising, but one cheap and easy method (with surprisingly good properties) is to take the maximal possible expected utility (the expected utility that person would get if the AI did exactly what they wanted) as 1, and the minimal possible expected utility (if the AI was to work completely against them) as 0.
Unfortunately, this utility function isn’t game-theoretically stable; if you expect your utility to be analyzed this way, you have an incentive to modify your utility function to clip or flatten the ends, to make your utility have a steeper gradient around the amount of utility you expect to receive.
That seems like it may be true of every scheme that doesn’t consider the causal origins of people’s utility functions. Does something like Gibbard-Satterthwaite apply?
Not specifically causal origins (there’s evolution), instead I suppose there might be a way of directly negating most effects resulting from such strategic considerations (that is, decisions that were influenced by their expected effect on the decision in question that wants to negate that effect).
“Easy” way of doing this: see what would have happened if everyone believed that you were using a model where liars don’t prosper (ie random dictator), but actually use a Pareto method.
Considering the causal origins of people’s utility functions is a nice hack, thanks for pointing it out! How far back do we need to go, though? Should my children benefit if I manipulate their utility function genetically while they’re in the womb?
Another way to aggregate utility functions is by simulated bargaining, but it’s biased in favor of rich and powerful people.
As far as needed to understand (the dependence of current agent’s values on (the dependence of (expected benefit from value extraction) on current agent’s values)). (Sorry, adding parens was simpler!)
This involves currently confusing “benefit” (to whom?) and assumed-mistaken “expected” (by whom?), supposedly referring to aspects of past agents (that built/determined the current agent) deciding on the strategic value bargaining. (As usual, ability to parse the world and see things that play the roles of elements of agents’ algorithms seems necessary to get anything of this sort done.)
If I’m rich it’s because I delayed consumption, allowing others to invest the capital that I had earned. Should we not allow these people some return on their investment?
To be clear, I’m not very sure the answer is yes; but nor do I think it’s clear that ‘wealth’ falls into the category of ‘things that should not influence CEV’, where things like ‘race’, ‘eye colour’ etc. live.
Fair point about delayed gratification, but you may also be rich because your parents were rich, or because you won the lottery, or because you robbed someone. Judging people by their bargaining power conflates all those possible reasons.
If you didn’t delay gratification and had expensive tastes, you’d spend the money quickly, regardless of how you got it.
Even if everyone did have expensive tastes, people who started off with less money would need to delay their gratification more. A very poor person might need to delay gratification an average of 80% of the time, since they couldn’t afford almost anything. A sufficiently rich person might only need to delay gratification 10% of the time without running into financial trouble. So if you wanted to reward delaying of gratification, then on average the poorer that a person was, the more you’d want to reward him
You can’t be Pareto and game-theoretically stable at the same time (I have a nice picture proof of that, that I’ll post some time). You can be stable without being Pareto—we each choose our favoured outcome, and go 50-50 between them. Then no one has an incentive to lie.
You can estimate where the others’ favoured outcomes and go a ways in the opposite direction to try to balance it out. Of course, if one of you takes this to the second level and the others are honest, then no one is happy except by coincidence (one of the honest people deviated from the mean more than you in the same way, and your overshoot happened to land on them).
Unfortunately, this utility function isn’t game-theoretically stable; if you expect your utility to be analyzed this way, you have an incentive to modify your utility function to clip or flatten the ends, to make your utility have a steeper gradient around the amount of utility you expect to receive.
That seems like it may be true of every scheme that doesn’t consider the causal origins of people’s utility functions. Does something like Gibbard-Satterthwaite apply?
Not specifically causal origins (there’s evolution), instead I suppose there might be a way of directly negating most effects resulting from such strategic considerations (that is, decisions that were influenced by their expected effect on the decision in question that wants to negate that effect).
“Easy” way of doing this: see what would have happened if everyone believed that you were using a model where liars don’t prosper (ie random dictator), but actually use a Pareto method.
Considering the causal origins of people’s utility functions is a nice hack, thanks for pointing it out! How far back do we need to go, though? Should my children benefit if I manipulate their utility function genetically while they’re in the womb?
Another way to aggregate utility functions is by simulated bargaining, but it’s biased in favor of rich and powerful people.
As far as needed to understand (the dependence of current agent’s values on (the dependence of (expected benefit from value extraction) on current agent’s values)). (Sorry, adding parens was simpler!)
This involves currently confusing “benefit” (to whom?) and assumed-mistaken “expected” (by whom?), supposedly referring to aspects of past agents (that built/determined the current agent) deciding on the strategic value bargaining. (As usual, ability to parse the world and see things that play the roles of elements of agents’ algorithms seems necessary to get anything of this sort done.)
If I’m rich it’s because I delayed consumption, allowing others to invest the capital that I had earned. Should we not allow these people some return on their investment?
To be clear, I’m not very sure the answer is yes; but nor do I think it’s clear that ‘wealth’ falls into the category of ‘things that should not influence CEV’, where things like ‘race’, ‘eye colour’ etc. live.
Fair point about delayed gratification, but you may also be rich because your parents were rich, or because you won the lottery, or because you robbed someone. Judging people by their bargaining power conflates all those possible reasons.
No; if you didn’t delay gratification you’d spend the money quickly, regardless of how you got it.
The funniest counterexample I know is Jefri Bolkiah =)
If you didn’t delay gratification and had expensive tastes, you’d spend the money quickly, regardless of how you got it.
Even if everyone did have expensive tastes, people who started off with less money would need to delay their gratification more. A very poor person might need to delay gratification an average of 80% of the time, since they couldn’t afford almost anything. A sufficiently rich person might only need to delay gratification 10% of the time without running into financial trouble. So if you wanted to reward delaying of gratification, then on average the poorer that a person was, the more you’d want to reward him
The same rich and powerful people who are most likely to be funding the research, maybe?
Today, to resolve their differences, people mostly just bargain I.R.L.
They do simulate bargains in their heads, but only to help them with the actual bargaining.
You can’t be Pareto and game-theoretically stable at the same time (I have a nice picture proof of that, that I’ll post some time). You can be stable without being Pareto—we each choose our favoured outcome, and go 50-50 between them. Then no one has an incentive to lie.
Edit: Picture-proof now posted at: http://lesswrong.com/r/discussion/lw/8qv/in_the_pareto_world_liars_prosper/
I seem to have an incentive to lie in that scenario.
You can estimate where the others’ favoured outcomes and go a ways in the opposite direction to try to balance it out. Of course, if one of you takes this to the second level and the others are honest, then no one is happy except by coincidence (one of the honest people deviated from the mean more than you in the same way, and your overshoot happened to land on them).