“A point I may not have made in these posts, but made in comments, is that the majority of humans today think that women should not have full rights, homosexuals should be killed or at least severely persecuted, and nerds should be given wedgies. These are not incompletely-extrapolated values that will change with more information; they are values. Opponents of gay marriage make it clear that they do not object to gay marriage based on a long-range utilitarian calculation; they directly value not allowing gays to marry. Many human values horrify most people on this list, so they shouldn’t be trying to preserve them.”
This has always been my principal objection to CEV. I strongly suspect that were it implemented, it would want the death of a lot of my friends, and quite possibly me, too.
Regarding CEV: My own worry is that lots of parts of human value get washed out as “incoherent”—whatever X is, if it isn’t a basic human biological drive, there are enough people out there that have different opinions on it to make CEV throw up its hands, declare it an “incoherent” desire, and proceed to leave it unsatisfied. As a result, CEV ends up saying that the best we can do is just make everyone a wirehead because pleasure is one of our few universal coherent desires while things like “self-determination” and “actual achievement in the real world” are a real mess to provide and barely make sense in the first place. Or something like that.
(Universal wireheading—with robots taking care of human bodies—at least serves as a lower bound on any proposed utopia; people, in general, really do want pleasure, even if they also want other things. See also “Reedspace’s Lower Bound”.)
I would like to see more discussion on the question of how we should distinguish between 1) things we value even at the expense of pleasure, and 2) things we mistakenly alieve are more pleasurable than pleasure.
Surely if there is something I will give up pleasure for, which I do not experience as pleasurable, that’s strong evidence that it is an example of 1 and not 2?
Yes, but there are other cases. If you prefer eating a cookie to having the pleasure centers in your brain maximally stimulated, are you sure that’s not because eating a cookie sounds on some level like it would be more pleasurable?
That’s a little unfair to the concept of CEV. If irreconcilable value conflicts persist after coherent extrapolation, I would think that a CEV function would output nothing, rather than using majoritarian analysis to resolve the conflict.
CEV is supposed to preserve those things that people value, and would continue to value were they more intelligent and better informed.
I value the lives of my friends. Many other people value the death of people like my friends. There is no reason to think that this is because they are less intelligent or less well-informed than me, as opposed to actually having different preferences.
TimS claimed that in a situation like that, CEV would do nothing, rather than impose the extrapolated will of the majority.
My claim is that there is nothing—not one single thing—which would be a value held by every person in the world, even were they more intelligent and better informed. An intelligent, informed psychopath has utterly different values from mine, and will continue to have utterly different values upon reflection. The CEV therefore either has to impose the majority preferences upon the minority, or do nothing at all.
There is no reason to think that this is because they are less intelligent or less well-informed than me, as opposed to actually having different preferences.
There are lots of reasons to think so. For example, they might want the death of your friends because they mistakenly believe that a deity exists.
Or for any number of other, non-religious reasons. And it could well be that extrapolating those people’s preferences would lead, not to them rejecting their beliefs, but to them wishing to bring their god into existence.
Either people have fundamentally different, irreconcilable, values or they don’t. If they do, then the argument I made is valid. If they don’t, then CEV(any random person) will give exactly the same result as CEV(humanity).
That means that either calculating CEV(humanity) is an unnecessary inefficiency, or CEV(humanity) will do nothing at all, or CEV(humanity) would lead to a world that is intolerable for at least some minority of people. I actually doubt that any of the people from the SI would disagree with that (remember the torture vs flyspecks argument).
That may be considered a reasonable tradeoff by the developers of an “F”AI, but it gives those minority groups to whom the post-AI world would be inimical equally rational reasons to oppose such a development.
As someone who does not believe in moral realism, I agree that CEV over all humans who ever lived (excluding sociopaths and such) will not output anything.
But I think that a moral realist should believe that CEV will output some value system, and that the produced value system will be right.
Edit2: It appears that copying and pasting from some places includes “http” even when my browser address doesn’t. But I did something wrong when copying from the philosophy dictionary.
I agree—assuming that CEV didn’t impose a majority view on a minority. My understanding of the SI’s arguments (and it’s only my understanding) is that they believe it will impose a majority view on a minority, but that they think that would be the right thing to do—that if the choice were beween 3^^^3 people getting a dustspeck in the eye or one person getting tortured for fifty years, the FAI would always make a choice, and that choice would be for the torture rather than the dustspecks.
Now, this may well be, overall, the rational choice to make as far as humanity as a whole goes, but it would most definitely not be the rational choice for the person who was getting tortured to support it.
And since, as far as I can see, most people only value a very small subset of humanity who identify as belonging to the same groups as them, I strongly suspect that in the utilitarian calculations of a “friendly” AI programmed with CEV, they would end up in the getting-tortured group, rather than the avoiding-dustspecks one.
that if the choice were beween 3^^^3 people getting a dustspeck in the eye or one person getting tortured for fifty years, the FAI would always make a choice, and that choice would be for the torture rather than the dustspecks
That is an entirely separate issue. If CEV(everyone) outputted a moral theory that held utility was additive, then the AI implementing it would choose torture over specks. In other words, utilitarians are committed to believing that specks is the wrong choice.
But there is no guarantee that CEV will output a utilitarian theory, even if you believe it will output something. SI (Eliezer, at least) believes CEV will output a utilitarian theory because SI believes utilitarian theories are right. But everyone agrees that “whether CEV will output something” is a different issue than “what CEV will output.”
Personally, I suspect CEV(everyone in the United States) would output something deotological—and might even output something that would pick specks. Again, assuming it outputs anything.
Either people have fundamentally different, irreconcilable, values or they don’t. If they do, then the argument I made is valid. If they don’t, then CEV(any random person) will give exactly the same result as CEV(humanity).
This is a false dilemma. If people have some values that are the same or reconcilable, then you will get different output from CEV(any random person) and CEV(humanity).
And note that an actual move by virtue ethicists is to exclude sociopaths from “humanity”.
TimS claimed that in a situation like that, CEV would do nothing, rather than impose the extrapolated will of the majority.
I agree with you in general, and want to further point out that there is no such thing as “doing nothing”. If doing nothing tends to allow your friends to continue living (because they have the power to defend themselves in the status quo), that is favoring their values. If doing nothing tends to allow your friends to be killed (because they are a powerless, persecuted minority in the status quo) that is favoring the other people’s values.
Of course, a lot depends on what we’re willing to consider a minority as opposed to something outside the set of things being considered at all.
E.g., I’m in a discussion elsethread with someone who I think would argue that if we ran CEV on the set of things capable of moral judgments, it would not include psychopaths in the first place, because psychopaths are incapable of moral judgments.
I disagree with this on several levels, but my point is simply that there’s an implicit assumption in your argument that terms like “person” have shared referents in this context, and I’m not sure they do.
In which case we wouldn’t be talking about CEV(humanity) but CEV(that subset of humanity which already share our values), where “our values” in this case includes excluding a load of people from humanity before you start. Psychopaths may or may not be capable of moral judgements, but they certainly have preferences, and would certainly find living in a world where all their preferences are discounted as intolerable as the rest of us would find living in a world where only their preferences counted.
I agree that psychopaths have preferences, and would find living in a world that anti-implemented their preferences intolerable.
In which case we wouldn’t be talking about CEV(humanity) but CEV(that subset of humanity which already share our values),
If you mean to suggest that the fact that the former phrase gets used in place of the latter is compelling evidence that we all agree about who to include, I disagree.
If you mean to suggest that it would be more accurate to use the latter phrase when that’s what we mean, I agree.
Ditto “CEV(that set of preference-havers which value X, Y, and Z)”.
I hope that everyone who discusses CEV understands that a very hard part of building a CEV function would be defining the criteria for inclusion in the subset of people whose values are considered. It’s almost circular, because figuring out who to exclude as “insufficiently moral” almost inherently requires the output of a CEV-like function to process.
I’m not sure I understand the question. In reference to the sociopath issue, I think it is clearer to say: (1) “I don’t want sociopaths (and the like) in the subset from which CEV is drawn” than to say that (2) “CEV is drawn from all humanity but sociopaths are by definition not human.”
Nonetheless, I don’t think (1) and (2) are different in any important respect. They just define key terms differently in order to say the same thing. In a rational society, I suspect it would make no difference, but in the current human society, ways words can be wrong makes (2) likely to lead to errors of reasoning.
Sorry, I’m being unclear. Let me try again. For simplicity, let us say that T(x) = TRUE if x is sufficiently moral to include in CEV, and FALSE otherwise. (I don’t mean to posit that we’ve actually implemented such a test.)
I’m asking if you mean to distinguish between: (1) CEV includes x where T(x) = TRUE and x is human, and (2) CEV includes x where T(x) = TRUE
I’m still not sure I understand the question. That said, there are two issues here.
First, I would expect CEV(Klingon) to output something if CEV(human) does, but I’m not aware of any actual species that I would expect CEV(non-human species) to output for. If such a species existed (i.e. CEV(dolphins) outputs a morality), I would advocate strongly for something very like equal rights between humans and dolphins.
But even in that circumstance, I would be very surprised if CEV(all dolphins & all humans) outputted something other than “Humans, do CEV(humanity). Dolphins, do CEV(dolphin)”
Of course, I don’t expect CEV(all of humanity ever) to output because I reject moral realism.
‘Coherent’ in CEV means that it makes up a coherent value system for all of humanity. By definition that means that there will be no value conflicts in CEV. But it does not mean that you will necessarily like it.
Why do we need a single CEV value system? A FAI can calculate as many value systems as it needs and keep incompatible humans separate. Group size is just another parameter to optimize. Religious fundamentalists can live in their own simulated universe, liberals in another.
Upvoting back to zero because I think this is an important question to address.
If I prefer that people not be tortured, and that’s more important to me than anything else, then I ought not prefer a system that puts all the torturers in their own part of the world where I don’t have to interact with them over a system that prevents them from torturing.
More generally, this strategy only works if there’s nothing I prefer/antiprefer exist, but merely things that I prefer/antiprefer to be aware of.
[T]here’s nothing I prefer/antiprefer exist, but merely things that I prefer/antiprefer to be aware of.
is a conceivable extrapolation from a starting point where you antiprefer something’s existence (in the extreme, with MWI you may not have much say what does/doesn’t exist, just how much of it in which branches).
It’s also possible that you hold both preferences (prefer X not exist, prefer not to be aware of X) and the existence preference gets dropped for being incompatible with other values held by other people while the awareness preference does not.
Assuming 100% isolation it would be indistinguishable from living in a universe where the Many Worlds Interpretation is true, but it still seems wrong. The FAI could consider avoiding groups whose even theoretical existence could cause offence, but I don’t see any good way to assign weight to this optimization pressure.
Even so, I think splitting humanity into multiple groups is likely to be a better outcome than a single group. I don’t consider the “failed utopia” described in http://lesswrong.com/lw/xu/failed_utopia_42/ to be particularly bad.
The failed utopia is better than our current world, certainly. But the genie isn’t Friendly.
In principle, I could interact with the immoral cluster. AI’s interference is not relevant to the morality of the situation because I was part of the creation of the AI. Otherwise, I would be morally justified in ignoring the suffering in some distant part of the world because it will have no practical impact on my life. By contrast, I simply cannot interact with other branches under the MWI—it’s a baked in property of the universe that I never had any input into.
Um, if you would object to your friends being killed (even if you knew more, thought faster, and grew up further with others), then it wouldn’t be coherent to value killing them.
Coherence is not a simple question of a majority vote. Coherence will reflect the balance, concentration, and strength of individual volitions. A minor, muddled preference of 60% of humanity might be countered by a strong, unmuddled preference of 10% of humanity. The variables are quantitative, not qualitative.
(...)
It should be easier to counter coherence than to create coherence.
(...)
In qualitative terms, our unimaginably alien, powerful, and humane future selves should have a strong ability to say “Wait! Stop! You’re going to predictably regret that!”, but we should require much higher standards of predictability and coherence before we trust the extrapolation that says “Do this specific positive thing, even if you can’t comprehend why.”
Though it’s not clear to me how the document would deal with Wei Dai’s point in the sibling comment. In the absence of coherence on the question of whether to protect, persecute, or ignore impopular minority groups, does CEV default to protecting them or ignoring them? You might say that as written, it would obviously not protect them, because there was no coherence in favor of doing so; but what if protection of minority groups is a side effect of other measures CEV was taking anyway?
(For what it’s worth, I suspect that extrapolation would in fact create enough coherence for this particular scenario not to be a problem.)
My understanding is that CEV is based on consensus, in which case the majority is meaningless.
If CEV doesn’t positively value some minority group not being killed (i.e., if it’s just indifferent due to not having a consensus), then the majority would be free to try to kill that group. So we really do need CEV to saying something about this, instead of nothing.
Wouldn’t CEV need to extract consensus values under a Rawlsian “veil of ignorance”?
It strikes me as very unlikely that there would be a consensus (or even majority) vote for killing gays or denying full rights to women under such a veil, because of the significant probability of ending up gay, and the more than 50% probability of being a woman. Prisons would be a lot better as well. The only reason illiberal values persist is because those who hold them know (or are confident) that they’re not personally going to be victims of them.
So CEV is either going to end up very liberal, or if done without the veil of ignorance, is not going to end up coherent at all. Sorry if that’s politics, the mind-killer.
True, and this could create some interesting choices for Rawlsians with very conservative values. Would they create a world with no gays, or no women? Would they do both???
Heinlein’s “Starship Troopers” discusses the death penalty imposed on a violent child rapist/murder. The narrator says there are two possibilities:
1) The killer was so deranged he didn’t know right from wrong. In that case, killing (or imprisoning him) is the only safe solution for the rest. Or, 2) The killer knew right from wrong, but couldn’t stop himself. Wouldn’t killing (or stopping) him be a favor, something he would want?
Why can’t that type of reasoning exist behind the veil of ignorance? Doesn’t it completely justify certain kinds of oppression? That said, there’s also an empirical question whether the argument applies to the particular group being oppressed.
Not dealing with your point, but that sort of analysis is why I find Heinlein so distasteful—the awful philosophy. For example in #1, 5 seconds of thought suffices to think of counterexamples like temporary derangements (drug use, treatable disease, particularly stressful circumstances, blows to the head), and more effort likely would turn up powerful empirical evidence like possibly an observation that most murderers do not murder again even after release (and obviously not execution).
Absolutely. What finally made me realize that Heinlein was not the bestest moral philosopher ever was noticing that all his books contained superheros—Stranger in a Strange Land is the best example. I’m not talking about the telekinetic powers, but the mental discipline. His moral theory might work for human-like creatures with perfect mental discipline, but for ordinary humans . . . not so much.
This was pretty common in sf of the early 20th century, actually — the trope of a special group of people with unusual mental disciplines giving them super powers and special moral status. See A. E. van Vogt (the Null-A books) or Doc Smith (the Lensman books) for other examples. There’s a reason Dianetics had so much success in the sf community of that era, I suspect — fans were primed for it.
Well, I’m not exactly a Heinlein scholar, but I’d say it shows up mainly in his late-period work, post Stranger in a Strange Land. Time Enough for Love and its sequels definitely qualify, but some of the stuff he’s most famous for—The Moon is a Harsh Mistress, Have Space Suit, Will Travel, et cetera—don’t seem to. Unfortunately, Heinlein’s reputation is based mainly on that later stuff.
The revolution in “Moon is a Harsh Mistress” cannot succeed without the aid of the supercomputer. That makes any moral philosophy implicit in that revolution questionable to the extent one asserts that the moral philosophy is true of humanity now.
To a lesser extend, “Starship Troopers” asserts that military service is a reliable way of screening for the kinds of moral qualities (like mental discipline) that make one trustworthy enough to be a high government official (or even to vote, if I recall correctly). In reality, those moral qualities are very thin on the ground in the real world, being much less common than suggested by the book. If the appropriate moral qualities were really that frequent, the sanity line would already be much high than it is.
I wouldn’t say the Starship Troopers government was fascist, but Heinlein clearly thinks they are doing things pretty well. The fact that the creation process of that government avoided fascism with no difficulty (it isn’t considered worth mentioning in the history) is precisely the lack of realism that I am criticizing.
What would a Rawlsian decider do? Institute a prison and psychiatric system, and some method of deciding between case 1 (psychiatric imprisonment to try and treat or at least prevent further harm) and case 2 (criminal imprisonment to deter like-minded people and prevent further harm from the killer/rapist). Also set up institutions for detecting and encouraging early treatment of child sex offenders before they moved to murder.
They would not want the death penalty in either case, nor would they want the prison/psychiatric system to be so appalling that they might prefer to be dead.
The Rawlsian would need to weigh the risk of being the raped/murdered child (or their parent) against the risk of being born with psychopathic or paedophile tendencies. If there was genuinely a significant deterrent from the death penalty, then the Rawlsian might accept it. But that looks unlikely in such cases.
Most of those who propose illiberal values do not do so under the presumption that they thereby harm the affected groups. A paternalistic attitude is much more common, and is not automatically inconsistent with preferences beyond a Rawlsian veil of ignorance.
An Omelasian attitude also seems consistent, for that matter, though even less likely.
As a matter of empirical fact, I think this is wrong. Men in sexist societies are really glad they’re not women (and even thank God they are not in some cases). They are likely to run in horror from the Rawlsian veil when they see the implications.
And anyway, isn’t that paternalism itself inconsistent with Rawlsian ignorance? Who would voluntarily accept a more than 50% chance of being treated like a patronized child (and a second-class citizen) for life?
And how is killing gays in the slightest bit a paternalistic attitude?
I’d never heard of Omelas, or anything like it.. so I doubt this will be part of CEV. Again, who would voluntarily accept the risk of being such a scapegoat, if it were an avoidable risk? (If it is not avoidable for some reason, then that is a fact that CEV would have to take into account, as would the Rawlsian choosers).
Who would voluntarily accept a more than 50% chance of being treated like a patronized child (and a second-class citizen) for life?
Someone believing that this sort of paternalism is essential to gender and unable or unwilling to accept a society without it. Someone convinced that this was part of God’s plan or otherwise metaphysically necessary. Someone not very fond of making independent decisions. I don’t think any of these categories are strikingly rare.
That’s about as specific as I’d like to get; anything more so would incur an unacceptable risk of political entanglements. In general, though, I think it’s important to distinguish fears and hatreds arising against groups which happen to be on the wrong side of some social line (and therefore identity) from the processes that led to that line being drawn in the first place: it’s possible, and IMO quite likely, for people to coherently support most traditional values concerning social dichotomies without coherently endorsing malice across them. This might not end up being stable, human psychology being what it is, but it doesn’t seem internally inconsistent.
The way people’s values intersect with the various consequences of their identities is quite complicated and I’m not sure I completely understand it, but I wouldn’t describe either as a subset of the other.
(Incidentally, around 51% of human births are male; more living humans are female but that’s because women live longer. This has absolutely no bearing on the argument, but it was bugging me.)
What you’ve described here is a person who would put adherence to an ideological system (or set of values derived from that system) above their own probable welfare. They would reason to themselves : yes my own personal welfare would probably be higher in an egalitarian society (or the risk of low personal welfare would be lower); but stuff that, I’m going to implement my current value system anyway. Even if it comes back to shoot me in the foot.
I agree that’s possible, but my impression is that very few humans would really want to do that. The tendency to put personal welfare first is enormous, and I really do believe that most of us would do that if choosing behind a Rawlsian veil.
What’s odd is that it is a classical conservative insight that human beings are mostly self-interested, and rather risk-adverse, and that society needs to be constructed to take that into account. It’s an insight I agree with by the way, and yet it is precisely this insight that leads to Rawlsian liberalism. Whereas to choose a different (conservative) value system, the choosers have to sacrifice their self-interest to that value system.
What you’ve described here is a person who would put adherence to an ideological system (or set of values derived from that system) above their own probable welfare.
Self-assessed welfare isn’t cleanly separable from ideology. People aren’t strict happiness maximizers; we value all sorts of abstract things, many of which are linked to the social systems and identity groups in which we’re embedded. Sometimes this ends up looking pretty irrational from the outside view, but from the inside giving them up would look unattractive for more or less the same reason that wireheading is unattractive to (most of) us.
Now, this does drift over time, both on a sort of random walk and in response to environmental pressures, which is what allows things like sexual revolutions to happen. During phase changes in this space, the affected social dichotomies are valued primarily in terms of avoiding social costs; that’s the usual time when they’re a salient issue instead of just part of the cultural background, and so it’s easy to imagine that that’s always what drives them. But I don’t think that’s the case; I think there’s a large region of value space where they really are treated as intrinsic to welfare, or as first-order consequences of intrinsic values.
Thanks again. I’m still not sure of the exact point you are making here, though.
Let’s take gender-based discrimination and unequal rights as a sample case. Are you arguing that someone wedded to an existing gender-biased value system would deliberately select a discriminatory society (over an equal rights one) even if they were choosing on the basis of self-interest? That they would fully understand that they have roughly 50% chance of getting the raw end of the deal, but still think that this deal would maximise their welfare overall?
I get the point that a committed ideologue could consciously decide here against self-interest. I’m less clear how someone could decide that way while still thinking it was in their self-interest. The only way I can make sense of such a decision is if were made on the basis of faulty understanding (i.e. they really can’t empathize very well, and think it would not be so bad after all to get born female in such a society).
In a separate post, I suggested a way that an AI could make the Rawlsian thought experiment real, by creating a simulated society to the deciders’ specifications, and then beaming them into roles in the simulation at random (via virtual reality/total immersion/direct neural interface or whatever). One variant to correct for faulty understanding might be to do it on an experimental basis. Once the choosers think they have made their minds up, they get beamed into a few randomly-selected folks in the sim, maybe for a few days or weeks (or years) at a time. After the experience of living in their chosen world for a while, in different places, times, roles etc. they are then asked if they want to change their mind. The AI will repeat until there is a stable preference, and then beam in permanently.
Returning to the root of the thread, the original objection to CEV was that most people alive today believe in unequal rights for women and essentially no rights for gays. The key question is therefore whether most people would really choose such a world in the Rawlsian set-up. And then, would most people continue to so-choose even after living in that world for a while in different roles?
If the answers are “no” then the Rawlsian veil of ignorance can remove this particular objection to CEV.
If they are “yes” then it cannot. Agreed?
That they would fully understand that they have roughly 50% chance of getting the raw end of the deal, but still think that this deal would maximise their welfare overall?
A lot of oppression of women seems to be justified by claims that if women aren’t second-class citizens, they won’t choose to have children, or at least not enough children for replacement. This makes women’s rights into an existential risk.
This argument also implies that societies and smaller groups where women have lower status and more children will out-breed and so eventually outcompete societies where women have equal rights. So people can also defend the lower status of women as a nationalistic or cultural self-defense impulse.
Are you arguing that someone wedded to an existing gender-biased value system would deliberately select a discriminatory society (over an equal rights one) even if they were choosing on the basis of self-interest? That they would fully understand that they have roughly 50% chance of getting the raw end of the deal, but still think that this deal would maximise their welfare overall?
Yes and no. Someone who’d internalized a discriminatory value system—who really believed in it, not just belief-in-belief, to use LW terminology—would interpret their self-interest and therefore their welfare in terms of that value system. They would be conscious of of what we would view as unequal rights, but would see these as neutral or positive on both sides, not as one “getting the raw end of the deal”—though they’d likely object to some of their operational consequences. This implies, of course, a certain essentialism, and only applies to certain forms of discrimination: recent top-down imposition of values isn’t stable in this way.
As a toy example, read 1 Corinthians 11, and try to think of the mentality implied by taking that as the literal word of God—not just advice from some vague authority, but an independent axiom of a value system backed by the most potent proofs imaginable. Applied to an egalitarian society, what would such a value system say about the (value-subjective) welfare of the women—or for that matter the men—in it?
the original objection to CEV was that most people alive today believe in unequal rights for women and essentially no rights for gays. The key question is therefore whether most people would really choose such a world in the Rawlsian set-up. And then, would most people continue to so-choose even after living in that world for a while in different roles?
This, on the other hand, is essentially an anthropology question. The answer depends on the extent of discriminatory traditional cultures, on the strength of belief in them, and on the commonalities between them: “unequal rights” isn’t a value, it’s a judgment call over a value system, and the specific unequal values that we object to may be quite different between cultures. I’m not an anthropologist, so I can’t really answer that question—but if I had to, I’d doubt that a reflectively stable consensus exists for egalitarianism or for any particular form of discrimination, with or without the Rawlsian wrinkle.
So this would be like the “separate but equal” argument? To paraphrase in a gender context: “Men and women are very different, and need to be treated differently under the law—both human and divine law. But it’s not like the female side is really worse off because of this different treatment”.
That—I think—would count as a rather basic factual misunderstanding of how discrimination really works. It ought to be correctable pretty damn fast by a trip into the simulator.
(Incidentally, I grew up in a fundamentalist church until my teens, and one of the things I remember clearly was the women and teen girls being very upset about being told that they had to shut up in church, or wear hats or long hair, or that they couldn’t be elders, or whatever. They also really hated having St Paul and the Corinthians thrown at them; the ones who believed in Bible inerrancy were sure the original Greek said something different, and that the sacred text was being misinterpreted and spun against them. Since it is an absolute precondition for an inerrantist position that correct interpretations are difficult, and up for grabs, this was no more unreasonable than the version spouted by the all-male elders.)
That—I think—would count as a rarher basic factual misunderstanding of how discrimination really works. It ought to be correctable pretty damn fast by a trip into the simulator.
Well, I won’t rule it out. But if you grow up in the West—even in one of its more traditionalist enclaves—that means you’ve grown up surrounded by some of the most fantastically egalitarian rhetoric the world’s ever generated, and I think one consequence of that is the adoption of a rather totalizing attitude toward any form of discrimination. Not that that’s a bad thing; discrimination’s bad news. But it does make it kind of hard to grok stratified social organization in any kind of unbiased way.
I grew up secular, albeit in one of the more conservative parts of my home state. But I have read a lot of social commentary from the Middle Ages and the Classical period, and I’ve visited a couple of highly traditionalist non-Western countries. Both seem to exhibit an attitude towards what we’d call unequal rights that’s pretty damned strange for those of us who were raised on Max Weber and Malcolm X, and I wouldn’t put the differences down to ignorance.
But I have read a lot of social commentary from the Middle Ages and the Classical period, and I’ve visited a couple of highly traditionalist non-Western countries.
Of course there is an enormous selection bias here. You’re reading the opinions of the tiny minority who were a) literate, b) had time to write social commentary, c) didn’t have their writings burned or otherwise censored and d) were preserved for later generations by copyists. It’s very difficult to tell whether they represented the CEV of their time (or anything like it). And on visiting other cultures, even in the present, I can only reflect that if you’d visited the fundie church of my childhood you’d have seen an overt culture of traditionalist paternalism/sexism, but wouldn’t have seen the genuine hurt and pain of the 50% or so who really wished it wasn’t like that. Being denied a public voice, you couldn’t have heard them. That’s kind of the point.
I’ve also visited a few non-Western countries in the world (on business) and to the extent the people there have voiced opinions about their situation versus ours (which was not very often), they’ve been rather keen to make their countries more like ours in terms of liberty, equality and the pursuit of shed loads of money.. Or leave for the West if they can. Sheer poverty sucks too.
The obvious (paternalistic) answer is that they believe that, conditioned on them being born female, their self-interest is improved by paternalistic treatment of all women vs equality.
In order to convince them otherwise, you would (at a minimum) have to run multiple world sims, not just multiple placements in one world. You would also have to forcibly give them sufficiently rational thought processes that they could interpret the evidence you forced upon them. I’m not sure that forcibly messing with people’s thought processes is ethical, or that you could really claim it was a coherent extrapolation after you had performed that much involuntary mind surgery on them.
In order to convince them otherwise, you would (at a minimum) have to run multiple world sims, not just multiple placements in one world. You would also have to forcibly give them sufficiently rational thought processes that they could interpret the evidence you forced upon them
Disagree. A simple classroom lesson is often sufficient to get the point across:
I’ve met women who honestly and persistently profess that women should not be allowed to vote. In at least one case, even in private, to a person they really want to like them and who very clearly disagrees with them.
That doesn’t surprise me… I’ve had the same experience once or twice, in mixed company, and with strong feminists in the room. The subsequent conversations along the lines of “But women chained themselves to railings, and threw themselves under horses to get the vote; how can you betray them like that?” were quite amusing. Especially when followed by the retort “Well I’ve got a right to my own opinion just as much as anyone else—surely you respect that as a feminist!”
I’ve also met quite a few people who think that no-one should vote. (“If it did any good, it would have been abolished years ago” a position I have a lot more sympathy for these days than I ever used to).
My preferred society (in a Rawlsian setting) might not actually have much voting at all, except on key constitutional issues. State and national political offices (parliaments, presidents etc) would be filled at random (in an analogue to jury service) and for a limited time period. After the victims had passed a few laws and a budget, they would be allowed to go home again. No-one would give a damn about gaffes, going off message, or the odd sex scandal, because it would happen all the time, and have very limited impact. I think there would also need to be mandatory citizen service on boring committees, local government roles, planning permission and drainage enquiries etc to stop professional civil servants, lobbyists or wonks ruling the roost: the necessary tedium would be considered part of everyone’s civic duty. This—in my opinion—is probably the biggest problem with politics. Much of it is so dull, or soul-destroying, that no-one with any sense wants to do it, so it is left to those without any sense.
The only reason illiberal values persist is because those who hold them know (or are confident) that they’re not personally going to be victims of them.
You might be right, but I’m less sure of this.
Someone with more historical or anthropological knowledge than I is welcome to correct me, but I’m given to understand that many of those whom we would consider victims of an oppressive social system, actually support the system. (E.g., while woman’s suffrage seems obvious now, there were many female anti-suffragists at the time.) It’s likely that such sentiments would be nullified by a “knew more, thought faster, &c.” extrapolation, but I don’t want to be too confident about the output of an algorithm that is as yet entirely hypothetical.
Furthermore, the veil of ignorance has its own problems: what does it mean for someone to have possibly been someone else? To illustrate the problem, consider an argument that might be made by (our standard counterexample) a hypothetical agent who wants only to maximize the number of paperclips in the universe:
The only reason non-paperclip-maximizing values persist is because those who hold them know (or are confident) that they’re not personally going to be victims of them (because they already know that they happened to have been born as humans rather than paperclip-maximizers).
---which does not seem convincing. Of course, humans in oppressed groups and humans in privileged groups are inexpressibly more similar to each other than humans are to paperclip-maximizers, but I still think this thought experiment highlights a methodological issue that proponents of a veil of ignorance would do well to address.
Someone with more historical or anthropological knowledge than I is welcome to correct me, but I’m given to understand that many of those whom we would consider victims of an oppressive social system, actually support the system.
Isn’t the main evidence that victims of oppressive social systems want to escape from them at every opportunity? There are reasons for refugees, and reasons that the flows are in consistent directions.
And if anti-suffragism had been truly popular, then having got the vote, women would have immediately voted to take it away again. Does this make sense?
Some other points:
CEV is about human values, and human choices, rather than paper-clippers. I doubt we’d get a CEV across wildly-different utility functions in the first place.
I’m happy to admit that CEV might not exist in the veil of ignorance case either, but it seems more likely to.
I’m getting a few down-votes here. Is the general consensus here that this is too close to politics, and that is a taboo subject (as it is a mind-killer)? Or is the “veil of ignorance” idea not an important part of CEV?
Just because some despised minorities exist today, doesn’t mean they will continue to exist in the future under CEV. If a big enough majority clearly wishes that “no members of that group continue to exist” (e.g. kill existing gays AND no new ones ever to be born), then the CEV may implement that, and the veil of ignorance won’t change this, because you can’t be ignorant about being a minority member in a future where no-one is.
Perhaps, but I think my point stands. CEV will use a veil of ignorance, or it won’t be coherent. It may be incoherent with the veil as well, but I doubt it. Real human beings look after number one much more than they’d ever care to admit, and won’t take stupid risks when choosing under the veil.
One very intriguing thought about an AI is that it could make the Rawlsian choice a real one. Create a simulated society to the choosers’ preferences, and then beam them in at random...
“A point I may not have made in these posts, but made in comments, is that the majority of humans today think that women should not have full rights, homosexuals should be killed or at least severely persecuted, and nerds should be given wedgies. These are not incompletely-extrapolated values that will change with more information; they are values. Opponents of gay marriage make it clear that they do not object to gay marriage based on a long-range utilitarian calculation; they directly value not allowing gays to marry. Many human values horrify most people on this list, so they shouldn’t be trying to preserve them.”
This has always been my principal objection to CEV. I strongly suspect that were it implemented, it would want the death of a lot of my friends, and quite possibly me, too.
Regarding CEV: My own worry is that lots of parts of human value get washed out as “incoherent”—whatever X is, if it isn’t a basic human biological drive, there are enough people out there that have different opinions on it to make CEV throw up its hands, declare it an “incoherent” desire, and proceed to leave it unsatisfied. As a result, CEV ends up saying that the best we can do is just make everyone a wirehead because pleasure is one of our few universal coherent desires while things like “self-determination” and “actual achievement in the real world” are a real mess to provide and barely make sense in the first place. Or something like that.
(Universal wireheading—with robots taking care of human bodies—at least serves as a lower bound on any proposed utopia; people, in general, really do want pleasure, even if they also want other things. See also “Reedspace’s Lower Bound”.)
I would like to see more discussion on the question of how we should distinguish between 1) things we value even at the expense of pleasure, and 2) things we mistakenly alieve are more pleasurable than pleasure.
Surely if there is something I will give up pleasure for, which I do not experience as pleasurable, that’s strong evidence that it is an example of 1 and not 2?
Yes, but there are other cases. If you prefer eating a cookie to having the pleasure centers in your brain maximally stimulated, are you sure that’s not because eating a cookie sounds on some level like it would be more pleasurable?
I’m not sure how I could ever be sure of such a thing, but it certainly seems implausible to me.
That’s a little unfair to the concept of CEV. If irreconcilable value conflicts persist after coherent extrapolation, I would think that a CEV function would output nothing, rather than using majoritarian analysis to resolve the conflict.
Then since there is not one single value about which every single human being on the planet can agree, a CEV function would output nothing at all.
Tense confusion.
CEV is supposed to preserve those things that people value, and would continue to value were they more intelligent and better informed. I value the lives of my friends. Many other people value the death of people like my friends. There is no reason to think that this is because they are less intelligent or less well-informed than me, as opposed to actually having different preferences. TimS claimed that in a situation like that, CEV would do nothing, rather than impose the extrapolated will of the majority.
My claim is that there is nothing—not one single thing—which would be a value held by every person in the world, even were they more intelligent and better informed. An intelligent, informed psychopath has utterly different values from mine, and will continue to have utterly different values upon reflection. The CEV therefore either has to impose the majority preferences upon the minority, or do nothing at all.
There are lots of reasons to think so. For example, they might want the death of your friends because they mistakenly believe that a deity exists.
Or for any number of other, non-religious reasons. And it could well be that extrapolating those people’s preferences would lead, not to them rejecting their beliefs, but to them wishing to bring their god into existence.
Either people have fundamentally different, irreconcilable, values or they don’t. If they do, then the argument I made is valid. If they don’t, then CEV(any random person) will give exactly the same result as CEV(humanity).
That means that either calculating CEV(humanity) is an unnecessary inefficiency, or CEV(humanity) will do nothing at all, or CEV(humanity) would lead to a world that is intolerable for at least some minority of people. I actually doubt that any of the people from the SI would disagree with that (remember the torture vs flyspecks argument).
That may be considered a reasonable tradeoff by the developers of an “F”AI, but it gives those minority groups to whom the post-AI world would be inimical equally rational reasons to oppose such a development.
As someone who does not believe in moral realism, I agree that CEV over all humans who ever lived (excluding sociopaths and such) will not output anything.
But I think that a moral realist should believe that CEV will output some value system, and that the produced value system will be right.
In short, I think one’s belief about whether CEV will output something is isomorphic on whether one believes in [moral realism] (plato.stanford.edu/entries/moral-realism/).
Edit: link didn’t work, so separated it out.
Have you tried putting
http://
in front of the URL?(Edit: the backtick thing to show verbatim code isn’t working properly for some reason, but you know what I mean.)
moral realism.
Edit: Apparently that was the problem. Thanks.
Edit2: It appears that copying and pasting from some places includes “http” even when my browser address doesn’t. But I did something wrong when copying from the philosophy dictionary.
I agree—assuming that CEV didn’t impose a majority view on a minority. My understanding of the SI’s arguments (and it’s only my understanding) is that they believe it will impose a majority view on a minority, but that they think that would be the right thing to do—that if the choice were beween 3^^^3 people getting a dustspeck in the eye or one person getting tortured for fifty years, the FAI would always make a choice, and that choice would be for the torture rather than the dustspecks.
Now, this may well be, overall, the rational choice to make as far as humanity as a whole goes, but it would most definitely not be the rational choice for the person who was getting tortured to support it.
And since, as far as I can see, most people only value a very small subset of humanity who identify as belonging to the same groups as them, I strongly suspect that in the utilitarian calculations of a “friendly” AI programmed with CEV, they would end up in the getting-tortured group, rather than the avoiding-dustspecks one.
This is not clear.
That is an entirely separate issue. If CEV(everyone) outputted a moral theory that held utility was additive, then the AI implementing it would choose torture over specks. In other words, utilitarians are committed to believing that specks is the wrong choice.
But there is no guarantee that CEV will output a utilitarian theory, even if you believe it will output something. SI (Eliezer, at least) believes CEV will output a utilitarian theory because SI believes utilitarian theories are right. But everyone agrees that “whether CEV will output something” is a different issue than “what CEV will output.”
Personally, I suspect CEV(everyone in the United States) would output something deotological—and might even output something that would pick specks. Again, assuming it outputs anything.
This is a false dilemma. If people have some values that are the same or reconcilable, then you will get different output from CEV(any random person) and CEV(humanity).
And note that an actual move by virtue ethicists is to exclude sociopaths from “humanity”.
I agree with you in general, and want to further point out that there is no such thing as “doing nothing”. If doing nothing tends to allow your friends to continue living (because they have the power to defend themselves in the status quo), that is favoring their values. If doing nothing tends to allow your friends to be killed (because they are a powerless, persecuted minority in the status quo) that is favoring the other people’s values.
Of course, a lot depends on what we’re willing to consider a minority as opposed to something outside the set of things being considered at all.
E.g., I’m in a discussion elsethread with someone who I think would argue that if we ran CEV on the set of things capable of moral judgments, it would not include psychopaths in the first place, because psychopaths are incapable of moral judgments.
I disagree with this on several levels, but my point is simply that there’s an implicit assumption in your argument that terms like “person” have shared referents in this context, and I’m not sure they do.
In which case we wouldn’t be talking about CEV(humanity) but CEV(that subset of humanity which already share our values), where “our values” in this case includes excluding a load of people from humanity before you start. Psychopaths may or may not be capable of moral judgements, but they certainly have preferences, and would certainly find living in a world where all their preferences are discounted as intolerable as the rest of us would find living in a world where only their preferences counted.
I agree that psychopaths have preferences, and would find living in a world that anti-implemented their preferences intolerable.
If you mean to suggest that the fact that the former phrase gets used in place of the latter is compelling evidence that we all agree about who to include, I disagree.
If you mean to suggest that it would be more accurate to use the latter phrase when that’s what we mean, I agree.
Ditto “CEV(that set of preference-havers which value X, Y, and Z)”.
I definitely meant the second interpretation of that phrase.
I hope that everyone who discusses CEV understands that a very hard part of building a CEV function would be defining the criteria for inclusion in the subset of people whose values are considered. It’s almost circular, because figuring out who to exclude as “insufficiently moral” almost inherently requires the output of a CEV-like function to process.
How committed are you to the word “subset” here?
I’m not sure I understand the question. In reference to the sociopath issue, I think it is clearer to say:
(1) “I don’t want sociopaths (and the like) in the subset from which CEV is drawn”
than to say that
(2) “CEV is drawn from all humanity but sociopaths are by definition not human.”
Nonetheless, I don’t think (1) and (2) are different in any important respect. They just define key terms differently in order to say the same thing. In a rational society, I suspect it would make no difference, but in the current human society, ways words can be wrong makes (2) likely to lead to errors of reasoning.
Sorry, I’m being unclear. Let me try again.
For simplicity, let us say that T(x) = TRUE if x is sufficiently moral to include in CEV, and FALSE otherwise. (I don’t mean to posit that we’ve actually implemented such a test.)
I’m asking if you mean to distinguish between:
(1) CEV includes x where T(x) = TRUE and x is human, and
(2) CEV includes x where T(x) = TRUE
I’m still not sure I understand the question. That said, there are two issues here.
First, I would expect CEV(Klingon) to output something if CEV(human) does, but I’m not aware of any actual species that I would expect CEV(non-human species) to output for. If such a species existed (i.e. CEV(dolphins) outputs a morality), I would advocate strongly for something very like equal rights between humans and dolphins.
But even in that circumstance, I would be very surprised if CEV(all dolphins & all humans) outputted something other than “Humans, do CEV(humanity). Dolphins, do CEV(dolphin)”
Of course, I don’t expect CEV(all of humanity ever) to output because I reject moral realism.
I think that answers my question. Thanks.
‘Coherent’ in CEV means that it makes up a coherent value system for all of humanity. By definition that means that there will be no value conflicts in CEV. But it does not mean that you will necessarily like it.
Why do we need a single CEV value system? A FAI can calculate as many value systems as it needs and keep incompatible humans separate. Group size is just another parameter to optimize. Religious fundamentalists can live in their own simulated universe, liberals in another.
Upvoting back to zero because I think this is an important question to address.
If I prefer that people not be tortured, and that’s more important to me than anything else, then I ought not prefer a system that puts all the torturers in their own part of the world where I don’t have to interact with them over a system that prevents them from torturing.
More generally, this strategy only works if there’s nothing I prefer/antiprefer exist, but merely things that I prefer/antiprefer to be aware of.
It’s a potential outcome, I suppose, in that
is a conceivable extrapolation from a starting point where you antiprefer something’s existence (in the extreme, with MWI you may not have much say what does/doesn’t exist, just how much of it in which branches).
It’s also possible that you hold both preferences (prefer X not exist, prefer not to be aware of X) and the existence preference gets dropped for being incompatible with other values held by other people while the awareness preference does not.
The child molester cluster (where they grow child simply to molest them, then kill them) doesn’t bother you, even if you never interact with it?
Because I’m fairly certain I wouldn’t like what CEV(child molester) would output and wouldn’t want an AI to implement it.
Assuming 100% isolation it would be indistinguishable from living in a universe where the Many Worlds Interpretation is true, but it still seems wrong. The FAI could consider avoiding groups whose even theoretical existence could cause offence, but I don’t see any good way to assign weight to this optimization pressure.
Even so, I think splitting humanity into multiple groups is likely to be a better outcome than a single group. I don’t consider the “failed utopia” described in http://lesswrong.com/lw/xu/failed_utopia_42/ to be particularly bad.
Well, not if “child-molesters” and “non-child-molestors” are competing for limited resources.
The failed utopia is better than our current world, certainly. But the genie isn’t Friendly.
In principle, I could interact with the immoral cluster. AI’s interference is not relevant to the morality of the situation because I was part of the creation of the AI. Otherwise, I would be morally justified in ignoring the suffering in some distant part of the world because it will have no practical impact on my life. By contrast, I simply cannot interact with other branches under the MWI—it’s a baked in property of the universe that I never had any input into.
What if space travel turns out to be impossible, and the superintelligence has to allocate the limited computational resources of the solar system?
Um, if you would object to your friends being killed (even if you knew more, thought faster, and grew up further with others), then it wouldn’t be coherent to value killing them.
Just because I wouldn’t value that, doesn’t mean that the majority of the world wouldn’t. Which is my whole point.
My understanding is that CEV is based on consensus, in which case the majority is meaningless.
Some quotes from the CEV document:
Though it’s not clear to me how the document would deal with Wei Dai’s point in the sibling comment. In the absence of coherence on the question of whether to protect, persecute, or ignore impopular minority groups, does CEV default to protecting them or ignoring them? You might say that as written, it would obviously not protect them, because there was no coherence in favor of doing so; but what if protection of minority groups is a side effect of other measures CEV was taking anyway?
(For what it’s worth, I suspect that extrapolation would in fact create enough coherence for this particular scenario not to be a problem.)
Thank you. So, not quite consensus but similarly biased in favor if inaction.
If CEV doesn’t positively value some minority group not being killed (i.e., if it’s just indifferent due to not having a consensus), then the majority would be free to try to kill that group. So we really do need CEV to saying something about this, instead of nothing.
Assuming we have no other checks on behavior, yes. I’m not sure, pending more reflection, whether that’s a fair assumption or not...
There is absolutely no reason to think that the values of all humans, extrapolated in some way, will arrive at a consensus.
Wouldn’t CEV need to extract consensus values under a Rawlsian “veil of ignorance”?
It strikes me as very unlikely that there would be a consensus (or even majority) vote for killing gays or denying full rights to women under such a veil, because of the significant probability of ending up gay, and the more than 50% probability of being a woman. Prisons would be a lot better as well. The only reason illiberal values persist is because those who hold them know (or are confident) that they’re not personally going to be victims of them.
So CEV is either going to end up very liberal, or if done without the veil of ignorance, is not going to end up coherent at all. Sorry if that’s politics, the mind-killer.
Note that there’s nothing physically impossible about altering the probability of being born gay, straight, bi, male, female, asexual, etc.
True, and this could create some interesting choices for Rawlsians with very conservative values. Would they create a world with no gays, or no women? Would they do both???
I don’t know how to reply to this without violating the site’s proscription on discussions of politics, which I prefer not to do.
OK—the comment was pretty flippant anyway. Consider it withdrawn.
Heinlein’s “Starship Troopers” discusses the death penalty imposed on a violent child rapist/murder. The narrator says there are two possibilities:
1) The killer was so deranged he didn’t know right from wrong. In that case, killing (or imprisoning him) is the only safe solution for the rest. Or,
2) The killer knew right from wrong, but couldn’t stop himself. Wouldn’t killing (or stopping) him be a favor, something he would want?
Why can’t that type of reasoning exist behind the veil of ignorance? Doesn’t it completely justify certain kinds of oppression? That said, there’s also an empirical question whether the argument applies to the particular group being oppressed.
Not dealing with your point, but that sort of analysis is why I find Heinlein so distasteful—the awful philosophy. For example in #1, 5 seconds of thought suffices to think of counterexamples like temporary derangements (drug use, treatable disease, particularly stressful circumstances, blows to the head), and more effort likely would turn up powerful empirical evidence like possibly an observation that most murderers do not murder again even after release (and obviously not execution).
Absolutely. What finally made me realize that Heinlein was not the bestest moral philosopher ever was noticing that all his books contained superheros—Stranger in a Strange Land is the best example. I’m not talking about the telekinetic powers, but the mental discipline. His moral theory might work for human-like creatures with perfect mental discipline, but for ordinary humans . . . not so much.
This was pretty common in sf of the early 20th century, actually — the trope of a special group of people with unusual mental disciplines giving them super powers and special moral status. See A. E. van Vogt (the Null-A books) or Doc Smith (the Lensman books) for other examples. There’s a reason Dianetics had so much success in the sf community of that era, I suspect — fans were primed for it.
Is that true of all of Heinlein’s books? I would say that most of them (including Starship Troopers) don’t have superheroes.
Well, I’m not exactly a Heinlein scholar, but I’d say it shows up mainly in his late-period work, post Stranger in a Strange Land. Time Enough for Love and its sequels definitely qualify, but some of the stuff he’s most famous for—The Moon is a Harsh Mistress, Have Space Suit, Will Travel, et cetera—don’t seem to. Unfortunately, Heinlein’s reputation is based mainly on that later stuff.
The revolution in “Moon is a Harsh Mistress” cannot succeed without the aid of the supercomputer. That makes any moral philosophy implicit in that revolution questionable to the extent one asserts that the moral philosophy is true of humanity now.
To a lesser extend, “Starship Troopers” asserts that military service is a reliable way of screening for the kinds of moral qualities (like mental discipline) that make one trustworthy enough to be a high government official (or even to vote, if I recall correctly). In reality, those moral qualities are very thin on the ground in the real world, being much less common than suggested by the book. If the appropriate moral qualities were really that frequent, the sanity line would already be much high than it is.
It might be relevant to note that Heinlein served in the U.S. Navy before he was discharged due to medical reasons.
Most men in his generation did military service of some form.
I read Starship Troopers as a critique of fascism, not its endorsement, but I could be wrong...
I wouldn’t say the Starship Troopers government was fascist, but Heinlein clearly thinks they are doing things pretty well. The fact that the creation process of that government avoided fascism with no difficulty (it isn’t considered worth mentioning in the history) is precisely the lack of realism that I am criticizing.
Hmm, I could be confusing the book with the movie. I’ll need to re-read it again.
As long as we’re using sci-fi to inform our thinking on criminality and corrections, The Demolished Man is an interesting read.
What would a Rawlsian decider do? Institute a prison and psychiatric system, and some method of deciding between case 1 (psychiatric imprisonment to try and treat or at least prevent further harm) and case 2 (criminal imprisonment to deter like-minded people and prevent further harm from the killer/rapist). Also set up institutions for detecting and encouraging early treatment of child sex offenders before they moved to murder.
They would not want the death penalty in either case, nor would they want the prison/psychiatric system to be so appalling that they might prefer to be dead.
The Rawlsian would need to weigh the risk of being the raped/murdered child (or their parent) against the risk of being born with psychopathic or paedophile tendencies. If there was genuinely a significant deterrent from the death penalty, then the Rawlsian might accept it. But that looks unlikely in such cases.
Most of those who propose illiberal values do not do so under the presumption that they thereby harm the affected groups. A paternalistic attitude is much more common, and is not automatically inconsistent with preferences beyond a Rawlsian veil of ignorance.
An Omelasian attitude also seems consistent, for that matter, though even less likely.
As a matter of empirical fact, I think this is wrong. Men in sexist societies are really glad they’re not women (and even thank God they are not in some cases). They are likely to run in horror from the Rawlsian veil when they see the implications.
And anyway, isn’t that paternalism itself inconsistent with Rawlsian ignorance? Who would voluntarily accept a more than 50% chance of being treated like a patronized child (and a second-class citizen) for life?
And how is killing gays in the slightest bit a paternalistic attitude?
I’d never heard of Omelas, or anything like it.. so I doubt this will be part of CEV. Again, who would voluntarily accept the risk of being such a scapegoat, if it were an avoidable risk? (If it is not avoidable for some reason, then that is a fact that CEV would have to take into account, as would the Rawlsian choosers).
Someone believing that this sort of paternalism is essential to gender and unable or unwilling to accept a society without it. Someone convinced that this was part of God’s plan or otherwise metaphysically necessary. Someone not very fond of making independent decisions. I don’t think any of these categories are strikingly rare.
That’s about as specific as I’d like to get; anything more so would incur an unacceptable risk of political entanglements. In general, though, I think it’s important to distinguish fears and hatreds arising against groups which happen to be on the wrong side of some social line (and therefore identity) from the processes that led to that line being drawn in the first place: it’s possible, and IMO quite likely, for people to coherently support most traditional values concerning social dichotomies without coherently endorsing malice across them. This might not end up being stable, human psychology being what it is, but it doesn’t seem internally inconsistent.
The way people’s values intersect with the various consequences of their identities is quite complicated and I’m not sure I completely understand it, but I wouldn’t describe either as a subset of the other.
(Incidentally, around 51% of human births are male; more living humans are female but that’s because women live longer. This has absolutely no bearing on the argument, but it was bugging me.)
Thanks for the reply here, that was helpful.
What you’ve described here is a person who would put adherence to an ideological system (or set of values derived from that system) above their own probable welfare. They would reason to themselves : yes my own personal welfare would probably be higher in an egalitarian society (or the risk of low personal welfare would be lower); but stuff that, I’m going to implement my current value system anyway. Even if it comes back to shoot me in the foot.
I agree that’s possible, but my impression is that very few humans would really want to do that. The tendency to put personal welfare first is enormous, and I really do believe that most of us would do that if choosing behind a Rawlsian veil.
What’s odd is that it is a classical conservative insight that human beings are mostly self-interested, and rather risk-adverse, and that society needs to be constructed to take that into account. It’s an insight I agree with by the way, and yet it is precisely this insight that leads to Rawlsian liberalism. Whereas to choose a different (conservative) value system, the choosers have to sacrifice their self-interest to that value system.
Self-assessed welfare isn’t cleanly separable from ideology. People aren’t strict happiness maximizers; we value all sorts of abstract things, many of which are linked to the social systems and identity groups in which we’re embedded. Sometimes this ends up looking pretty irrational from the outside view, but from the inside giving them up would look unattractive for more or less the same reason that wireheading is unattractive to (most of) us.
Now, this does drift over time, both on a sort of random walk and in response to environmental pressures, which is what allows things like sexual revolutions to happen. During phase changes in this space, the affected social dichotomies are valued primarily in terms of avoiding social costs; that’s the usual time when they’re a salient issue instead of just part of the cultural background, and so it’s easy to imagine that that’s always what drives them. But I don’t think that’s the case; I think there’s a large region of value space where they really are treated as intrinsic to welfare, or as first-order consequences of intrinsic values.
Thanks again. I’m still not sure of the exact point you are making here, though.
Let’s take gender-based discrimination and unequal rights as a sample case. Are you arguing that someone wedded to an existing gender-biased value system would deliberately select a discriminatory society (over an equal rights one) even if they were choosing on the basis of self-interest? That they would fully understand that they have roughly 50% chance of getting the raw end of the deal, but still think that this deal would maximise their welfare overall?
I get the point that a committed ideologue could consciously decide here against self-interest. I’m less clear how someone could decide that way while still thinking it was in their self-interest. The only way I can make sense of such a decision is if were made on the basis of faulty understanding (i.e. they really can’t empathize very well, and think it would not be so bad after all to get born female in such a society).
In a separate post, I suggested a way that an AI could make the Rawlsian thought experiment real, by creating a simulated society to the deciders’ specifications, and then beaming them into roles in the simulation at random (via virtual reality/total immersion/direct neural interface or whatever). One variant to correct for faulty understanding might be to do it on an experimental basis. Once the choosers think they have made their minds up, they get beamed into a few randomly-selected folks in the sim, maybe for a few days or weeks (or years) at a time. After the experience of living in their chosen world for a while, in different places, times, roles etc. they are then asked if they want to change their mind. The AI will repeat until there is a stable preference, and then beam in permanently.
Returning to the root of the thread, the original objection to CEV was that most people alive today believe in unequal rights for women and essentially no rights for gays. The key question is therefore whether most people would really choose such a world in the Rawlsian set-up. And then, would most people continue to so-choose even after living in that world for a while in different roles?
If the answers are “no” then the Rawlsian veil of ignorance can remove this particular objection to CEV. If they are “yes” then it cannot. Agreed?
A lot of oppression of women seems to be justified by claims that if women aren’t second-class citizens, they won’t choose to have children, or at least not enough children for replacement. This makes women’s rights into an existential risk.
This argument also implies that societies and smaller groups where women have lower status and more children will out-breed and so eventually outcompete societies where women have equal rights. So people can also defend the lower status of women as a nationalistic or cultural self-defense impulse.
Yes and no. Someone who’d internalized a discriminatory value system—who really believed in it, not just belief-in-belief, to use LW terminology—would interpret their self-interest and therefore their welfare in terms of that value system. They would be conscious of of what we would view as unequal rights, but would see these as neutral or positive on both sides, not as one “getting the raw end of the deal”—though they’d likely object to some of their operational consequences. This implies, of course, a certain essentialism, and only applies to certain forms of discrimination: recent top-down imposition of values isn’t stable in this way.
As a toy example, read 1 Corinthians 11, and try to think of the mentality implied by taking that as the literal word of God—not just advice from some vague authority, but an independent axiom of a value system backed by the most potent proofs imaginable. Applied to an egalitarian society, what would such a value system say about the (value-subjective) welfare of the women—or for that matter the men—in it?
This, on the other hand, is essentially an anthropology question. The answer depends on the extent of discriminatory traditional cultures, on the strength of belief in them, and on the commonalities between them: “unequal rights” isn’t a value, it’s a judgment call over a value system, and the specific unequal values that we object to may be quite different between cultures. I’m not an anthropologist, so I can’t really answer that question—but if I had to, I’d doubt that a reflectively stable consensus exists for egalitarianism or for any particular form of discrimination, with or without the Rawlsian wrinkle.
So this would be like the “separate but equal” argument? To paraphrase in a gender context: “Men and women are very different, and need to be treated differently under the law—both human and divine law. But it’s not like the female side is really worse off because of this different treatment”.
That—I think—would count as a rather basic factual misunderstanding of how discrimination really works. It ought to be correctable pretty damn fast by a trip into the simulator.
(Incidentally, I grew up in a fundamentalist church until my teens, and one of the things I remember clearly was the women and teen girls being very upset about being told that they had to shut up in church, or wear hats or long hair, or that they couldn’t be elders, or whatever. They also really hated having St Paul and the Corinthians thrown at them; the ones who believed in Bible inerrancy were sure the original Greek said something different, and that the sacred text was being misinterpreted and spun against them. Since it is an absolute precondition for an inerrantist position that correct interpretations are difficult, and up for grabs, this was no more unreasonable than the version spouted by the all-male elders.)
Well, I won’t rule it out. But if you grow up in the West—even in one of its more traditionalist enclaves—that means you’ve grown up surrounded by some of the most fantastically egalitarian rhetoric the world’s ever generated, and I think one consequence of that is the adoption of a rather totalizing attitude toward any form of discrimination. Not that that’s a bad thing; discrimination’s bad news. But it does make it kind of hard to grok stratified social organization in any kind of unbiased way.
I grew up secular, albeit in one of the more conservative parts of my home state. But I have read a lot of social commentary from the Middle Ages and the Classical period, and I’ve visited a couple of highly traditionalist non-Western countries. Both seem to exhibit an attitude towards what we’d call unequal rights that’s pretty damned strange for those of us who were raised on Max Weber and Malcolm X, and I wouldn’t put the differences down to ignorance.
Of course there is an enormous selection bias here. You’re reading the opinions of the tiny minority who were a) literate, b) had time to write social commentary, c) didn’t have their writings burned or otherwise censored and d) were preserved for later generations by copyists. It’s very difficult to tell whether they represented the CEV of their time (or anything like it). And on visiting other cultures, even in the present, I can only reflect that if you’d visited the fundie church of my childhood you’d have seen an overt culture of traditionalist paternalism/sexism, but wouldn’t have seen the genuine hurt and pain of the 50% or so who really wished it wasn’t like that. Being denied a public voice, you couldn’t have heard them. That’s kind of the point.
I’ve also visited a few non-Western countries in the world (on business) and to the extent the people there have voiced opinions about their situation versus ours (which was not very often), they’ve been rather keen to make their countries more like ours in terms of liberty, equality and the pursuit of shed loads of money.. Or leave for the West if they can. Sheer poverty sucks too.
The obvious (paternalistic) answer is that they believe that, conditioned on them being born female, their self-interest is improved by paternalistic treatment of all women vs equality.
In order to convince them otherwise, you would (at a minimum) have to run multiple world sims, not just multiple placements in one world. You would also have to forcibly give them sufficiently rational thought processes that they could interpret the evidence you forced upon them. I’m not sure that forcibly messing with people’s thought processes is ethical, or that you could really claim it was a coherent extrapolation after you had performed that much involuntary mind surgery on them.
Disagree. A simple classroom lesson is often sufficient to get the point across:
http://www.uen.org/Lessonplan/preview.cgi?LPid=536
Discrimination REALLY sucks.
I’ve met women who honestly and persistently profess that women should not be allowed to vote. In at least one case, even in private, to a person they really want to like them and who very clearly disagrees with them.
That doesn’t surprise me… I’ve had the same experience once or twice, in mixed company, and with strong feminists in the room. The subsequent conversations along the lines of “But women chained themselves to railings, and threw themselves under horses to get the vote; how can you betray them like that?” were quite amusing. Especially when followed by the retort “Well I’ve got a right to my own opinion just as much as anyone else—surely you respect that as a feminist!”
I’ve also met quite a few people who think that no-one should vote. (“If it did any good, it would have been abolished years ago” a position I have a lot more sympathy for these days than I ever used to).
My preferred society (in a Rawlsian setting) might not actually have much voting at all, except on key constitutional issues. State and national political offices (parliaments, presidents etc) would be filled at random (in an analogue to jury service) and for a limited time period. After the victims had passed a few laws and a budget, they would be allowed to go home again. No-one would give a damn about gaffes, going off message, or the odd sex scandal, because it would happen all the time, and have very limited impact. I think there would also need to be mandatory citizen service on boring committees, local government roles, planning permission and drainage enquiries etc to stop professional civil servants, lobbyists or wonks ruling the roost: the necessary tedium would be considered part of everyone’s civic duty. This—in my opinion—is probably the biggest problem with politics. Much of it is so dull, or soul-destroying, that no-one with any sense wants to do it, so it is left to those without any sense.
Kill their bodies, save their souls.
You might be right, but I’m less sure of this.
Someone with more historical or anthropological knowledge than I is welcome to correct me, but I’m given to understand that many of those whom we would consider victims of an oppressive social system, actually support the system. (E.g., while woman’s suffrage seems obvious now, there were many female anti-suffragists at the time.) It’s likely that such sentiments would be nullified by a “knew more, thought faster, &c.” extrapolation, but I don’t want to be too confident about the output of an algorithm that is as yet entirely hypothetical.
Furthermore, the veil of ignorance has its own problems: what does it mean for someone to have possibly been someone else? To illustrate the problem, consider an argument that might be made by (our standard counterexample) a hypothetical agent who wants only to maximize the number of paperclips in the universe:
---which does not seem convincing. Of course, humans in oppressed groups and humans in privileged groups are inexpressibly more similar to each other than humans are to paperclip-maximizers, but I still think this thought experiment highlights a methodological issue that proponents of a veil of ignorance would do well to address.
Isn’t the main evidence that victims of oppressive social systems want to escape from them at every opportunity? There are reasons for refugees, and reasons that the flows are in consistent directions.
And if anti-suffragism had been truly popular, then having got the vote, women would have immediately voted to take it away again. Does this make sense?
Some other points:
CEV is about human values, and human choices, rather than paper-clippers. I doubt we’d get a CEV across wildly-different utility functions in the first place.
I’m happy to admit that CEV might not exist in the veil of ignorance case either, but it seems more likely to.
I’m getting a few down-votes here. Is the general consensus here that this is too close to politics, and that is a taboo subject (as it is a mind-killer)? Or is the “veil of ignorance” idea not an important part of CEV?
Just because some despised minorities exist today, doesn’t mean they will continue to exist in the future under CEV. If a big enough majority clearly wishes that “no members of that group continue to exist” (e.g. kill existing gays AND no new ones ever to be born), then the CEV may implement that, and the veil of ignorance won’t change this, because you can’t be ignorant about being a minority member in a future where no-one is.
Isn’t there substantial disagreement over whether the veil of ignorance is sufficient or necessary to justify a moral theory?
Edit: Or just read what Nornagest said
Perhaps, but I think my point stands. CEV will use a veil of ignorance, or it won’t be coherent. It may be incoherent with the veil as well, but I doubt it. Real human beings look after number one much more than they’d ever care to admit, and won’t take stupid risks when choosing under the veil.
One very intriguing thought about an AI is that it could make the Rawlsian choice a real one. Create a simulated society to the choosers’ preferences, and then beam them in at random...
Even with a veil of ignorance, people won’t make the same choices—people fall in different places on the risk aversion/reward-seeking spectrum.