I can see why you might feel that way, if this was just a technical flaw in CEV that can be fixed with a simple patch. But I’ve been having a growing suspicion that the main philosophical underpinning of CEV, namely preference utilitarianism, is seriously wrong, and this story was meant to offer more evidence in that vein.
Why should anyone choose aggregation of preference over a personal FAI, other than under explicit pressure? Whatever obligations you feel (as part of your preference, as opposed to as part of an imaginary game where you play fair), will be payed in full according to your personal preference. This explicit pressure to include other folks in the mix can only be exerted by those present, and presumably “in the know”, so there is no need to include the dead or potential future folk. Whatever sympathy you have for them, you’ll have ability to express through the personal FAI. The virtue of laziness in FAI design again (this time, moral laziness).
Why should anyone choose aggregation of preference over a personal FAI, other than under explicit pressure?
But that doesn’t explain why Eliezer is vehemently against any unequal weighting of volitions in CEV, such as the “geopolitical-power-weighted set of volitions” that Roko suggested might be necessary if major political powers got involved.
As far as I can tell, Eliezer’s actual motivations for wanting to build CEV of humanity instead of a personal FAI are:
It’s the fair thing to do (in some non-game-theoretic sense of fairness).
The CEV of humanity has a better chance to leading to a good outcome than his personal extrapolated volition. (See this comment.)
Personally I don’t think these reasons are particularly good, and my current position is close to yours and Roko’s. But the fact that Eliezer has stuck to his beliefs on this topic makes me wonder if we’re missing something.
A given person’s preference is one thing, but their mind is another. If we do have a personal preference on one hand, and a collection of many people’s preferences on the other, the choice is simple. But the people included in the preference extraction procedure are not the same thing as their preferences. We use a collection of people, not a collection of preferences.
It’s not obvious to me that my personal preference is best described by my own brain and not an extrapolation from as many people’s brains as possible. Maybe I want to calculate, but I’m personally a flawed calculator, as are all the others, each in its own way. By examining as many calculators as possible, I could glimpse a better picture of how correct calculation is done, than I could ever find by only examining myself.
I value what is good not because humans value what is good, and I value whatever I in particular value (as opposed to what other people value) not because it is I who values that. If looking at other people’s minds helps me to figure out what should be valued, then I should do that.
That’s one argument for extrapolating collective volition; however, it’s a simple argument, and I expect that whatever can be found from my mind alone should be enough to reliably present arguments such as this, and thus to decide to go through the investigation of other people if that’s necessary to improve understanding of what I value. Whatever moral flaws specific to my mind exist, shouldn’t be severe enough to destroy this argument, if it’s true, but the argument could also be false. If it’s false, then I lose by defaulting to the collective option, but if it’s true, delegating it to FAI seems like a workable plan.
At the same time, there are likely practical difficulties in getting my mind in particular as the preference source to FAI. If I can’t get my preference in particular, then as close to the common ground for humanity as I can get (a decision to which as many people as possible agree as much as possible) is better for me (by its construction: if it’s better for most of the humanity, it’s also better for me in particular).
A similar problem shows up in hedonic utilitarianism, or indeed in any case where your algorithm for determining what to do requires ‘counting people.’
CEV is not preference utilitarianism, or any other first-order ethical theory. Rather, preference utilitarianism is the sort of thing that might be CEV’s output.
Obviously CEV isn’t identical to preference utilitarianism, but CEV and preference utilitarianism have the following principles in common, which the hack exploits:
Give people what they want, instead of what you think is good for them.
If different people want different things, give each individual equal weight.
It seems clear that Eliezer got these ideas from preference utilitarianism, and they share some of the same flaws as a result.
Well, you always have game-theoretic mix option, nothing “seriously wrong” with that (and so with preference aggregation more broadly construed than CEV in particular), although it’s necessarily a worse outcome than a personal FAI.
Less Wrong is a partially moderated community blog that allows general authors to contribute posts as well as comments. Users vote posts and comments up and down (with code based on Reddit’s open source). “Promoted” posts (appearing on the front page) are chosen by the editors on the basis of substantive new content, clear argument, good writing, popularity, and importance.
We suggest submitting links with a short description. Recommended books should have longer descriptions. Links will not be promoted unless they are truly excellent—the “promoted” posts are intended as a filtered stream for the casual/busy reader.
As far as I can tell, most people vote along the lines of the promotion criteria.
There’s isn’t a formal definition as of yet, but ideally I’d like to see top-level posts satisfy the following criteria:
1) Too long or involved to be included in an open thread.
2) Of general interest to the LW community.
3) Contribute substantial new and interesting points.
4) Likely to generate wide-ranging discussion.
I have trouble seeing this post as meeting 1 or 4.
People also complained about the “AI in a box boxes you post”, which was a great post nearly identical in structure to this one. Few people read the open thread; good posts should not default to the open thread. Why is your criteria for top-level posts so arbitrarily difficult? We are not facing a problem of an influx of low quality content and the moderation+promotion system works well.
My criteria for top-level posts is not “so arbitrarily difficult.” Frankly, I’m not completely sure that that the AI boxing you post should have been a top-level post either. However, given that that post did not focus on any specific AI solution but a more general set of issues, whereas this one focuses on CEV, there may be a distinction between them. That said, I agree that as of right now, the moderation/promotion system is working well. But I suspect that that is partially due to people implicitly applying criteria like the ones I listed in their moderation decisions.
Incidentally, I’m curious what evidence you have that the open threads are not as read as top-level posts. In particular, I’m not sure this applies to non-promoted top-level posts. I suspect that it is true, and indeed, if it isn’t then my own logic for wanting criterion 2 becomes substantially weaker. Now that you’ve made our shared premise explicit I have to wonder what evidence we have for the claim.
That’s an interesting point, but I’m having trouble seeing it as worthy of a top-level post. Maybe if you had a solution proposed also.
I can see why you might feel that way, if this was just a technical flaw in CEV that can be fixed with a simple patch. But I’ve been having a growing suspicion that the main philosophical underpinning of CEV, namely preference utilitarianism, is seriously wrong, and this story was meant to offer more evidence in that vein.
Why should anyone choose aggregation of preference over a personal FAI, other than under explicit pressure? Whatever obligations you feel (as part of your preference, as opposed to as part of an imaginary game where you play fair), will be payed in full according to your personal preference. This explicit pressure to include other folks in the mix can only be exerted by those present, and presumably “in the know”, so there is no need to include the dead or potential future folk. Whatever sympathy you have for them, you’ll have ability to express through the personal FAI. The virtue of laziness in FAI design again (this time, moral laziness).
But that doesn’t explain why Eliezer is vehemently against any unequal weighting of volitions in CEV, such as the “geopolitical-power-weighted set of volitions” that Roko suggested might be necessary if major political powers got involved.
As far as I can tell, Eliezer’s actual motivations for wanting to build CEV of humanity instead of a personal FAI are:
It’s the fair thing to do (in some non-game-theoretic sense of fairness).
The CEV of humanity has a better chance to leading to a good outcome than his personal extrapolated volition. (See this comment.)
Personally I don’t think these reasons are particularly good, and my current position is close to yours and Roko’s. But the fact that Eliezer has stuck to his beliefs on this topic makes me wonder if we’re missing something.
A given person’s preference is one thing, but their mind is another. If we do have a personal preference on one hand, and a collection of many people’s preferences on the other, the choice is simple. But the people included in the preference extraction procedure are not the same thing as their preferences. We use a collection of people, not a collection of preferences.
It’s not obvious to me that my personal preference is best described by my own brain and not an extrapolation from as many people’s brains as possible. Maybe I want to calculate, but I’m personally a flawed calculator, as are all the others, each in its own way. By examining as many calculators as possible, I could glimpse a better picture of how correct calculation is done, than I could ever find by only examining myself.
I value what is good not because humans value what is good, and I value whatever I in particular value (as opposed to what other people value) not because it is I who values that. If looking at other people’s minds helps me to figure out what should be valued, then I should do that.
That’s one argument for extrapolating collective volition; however, it’s a simple argument, and I expect that whatever can be found from my mind alone should be enough to reliably present arguments such as this, and thus to decide to go through the investigation of other people if that’s necessary to improve understanding of what I value. Whatever moral flaws specific to my mind exist, shouldn’t be severe enough to destroy this argument, if it’s true, but the argument could also be false. If it’s false, then I lose by defaulting to the collective option, but if it’s true, delegating it to FAI seems like a workable plan.
At the same time, there are likely practical difficulties in getting my mind in particular as the preference source to FAI. If I can’t get my preference in particular, then as close to the common ground for humanity as I can get (a decision to which as many people as possible agree as much as possible) is better for me (by its construction: if it’s better for most of the humanity, it’s also better for me in particular).
If that was your point, I wish you had gone into more detail about that in a top-level article.
A similar problem shows up in hedonic utilitarianism, or indeed in any case where your algorithm for determining what to do requires ‘counting people.’
Time inconsistency doesn’t bother me at all. It’s not my fault if you’re dead.
CEV is not preference utilitarianism, or any other first-order ethical theory. Rather, preference utilitarianism is the sort of thing that might be CEV’s output.
Obviously CEV isn’t identical to preference utilitarianism, but CEV and preference utilitarianism have the following principles in common, which the hack exploits:
Give people what they want, instead of what you think is good for them.
If different people want different things, give each individual equal weight.
It seems clear that Eliezer got these ideas from preference utilitarianism, and they share some of the same flaws as a result.
Whether preference utilitarianism is “right” or “wrong” would appear to depend on whether you are a preference utilitarian—or not.
Well, you always have game-theoretic mix option, nothing “seriously wrong” with that (and so with preference aggregation more broadly construed than CEV in particular), although it’s necessarily a worse outcome than a personal FAI.
To be honest, I loke that it’s straight and to the point without being stretched out. And the point itself is quite powerful.
I missed the part where Less Wrong had a definition of the required worthiness of a top-level post.
The definition is still being established, largely through comments such as JoshuaZ’s
From the About page:
As far as I can tell, most people vote along the lines of the promotion criteria.
There’s isn’t a formal definition as of yet, but ideally I’d like to see top-level posts satisfy the following criteria:
1) Too long or involved to be included in an open thread. 2) Of general interest to the LW community. 3) Contribute substantial new and interesting points. 4) Likely to generate wide-ranging discussion.
I have trouble seeing this post as meeting 1 or 4.
People also complained about the “AI in a box boxes you post”, which was a great post nearly identical in structure to this one. Few people read the open thread; good posts should not default to the open thread. Why is your criteria for top-level posts so arbitrarily difficult? We are not facing a problem of an influx of low quality content and the moderation+promotion system works well.
My criteria for top-level posts is not “so arbitrarily difficult.” Frankly, I’m not completely sure that that the AI boxing you post should have been a top-level post either. However, given that that post did not focus on any specific AI solution but a more general set of issues, whereas this one focuses on CEV, there may be a distinction between them. That said, I agree that as of right now, the moderation/promotion system is working well. But I suspect that that is partially due to people implicitly applying criteria like the ones I listed in their moderation decisions.
Incidentally, I’m curious what evidence you have that the open threads are not as read as top-level posts. In particular, I’m not sure this applies to non-promoted top-level posts. I suspect that it is true, and indeed, if it isn’t then my own logic for wanting criterion 2 becomes substantially weaker. Now that you’ve made our shared premise explicit I have to wonder what evidence we have for the claim.