Does Your Mor­al­ity Care What You Think?

Fol­lowup to: Math is Sub­junct­ively Ob­ject­ive, The Moral Void, Is Mor­al­ity Given?

Thus I re­call the study, though I can­not re­call the cita­tion:

Chil­dren, at some re­l­at­ively young age, were found to dis­tin­guish between:

  • The teacher, by say­ing that we’re al­lowed to stand on our desks, can make it right to do so.

  • The teacher, by say­ing that I’m al­lowed to take some­thing from an­other child’s back­pack, can­not make it right to do so.

Obert: “Well, I don’t know the cita­tion, but it sounds like a fas­cin­at­ing study. So even chil­dren, then, real­ize that moral facts are givens, bey­ond the abil­ity of teach­ers or par­ents to al­ter.”

Subhan: “You say that like it’s a good thing. Chil­dren may also think that people in Aus­tralia have to wear heavy boots from fall­ing off the other side of the Earth.”

Obert: “Call me Peter Pan, then, be­cause I never grew up on this one. Of course it doesn’t mat­ter what the teacher says. It doesn’t mat­ter what I say. It doesn’t even mat­ter what I think. Steal­ing is wrong. Do you dis­agree?

Subhan: “You don’t see me pick­ing your pock­ets, do you? Isn’t it enough that I choose not to steal from you—do I have to pre­tend it’s the law of the uni­verse?”

Obert: “Yes, or I can’t trust your com­mit­ment.”

Subhan: “A… re­veal­ing re­mark. But really, I don’t think that this ex­per­i­mental res­ult seems at all con­fus­ing, in light of the re­cent dis­cus­sion of sub­junct­ive ob­jectiv­ity—a dis­cus­sion in which Eliezer strongly sup­por­ted my po­s­i­tion, by the way.”

Obert: “Really? I thought Eliezer was fi­nally com­ing out in fa­vor of my po­s­i­tion.”

Subhan: “Huh? How do you get that?”

Obert: “The whole sub­text of ‘Math is Sub­junct­ively Ob­ject­ive’ is that mor­al­ity is just like math! Sure, we com­pute mor­al­ity in­side our own brains—where else would we com­pute it? But just be­cause we com­pute a quant­ity in­side our own brains, doesn’t mean that what is com­puted has a de­pend­ency on our own state of mind.”

Subhan: “I think we must have been read­ing dif­fer­ent Over­com­ing Bias posts! The whole sub­text of ‘Math is Sub­junct­ively Ob­ject­ive’ is to ex­plain away why mor­al­ity seems ob­ject­ive—to show that the feel­ing of a fixed given can arise without any ex­ternal ref­er­ent. When you ima­gine your­self think­ing that killing is right, your brain-that-ima­gines hasn’t yet been altered, so you carry out that moral ima­gin­a­tion with your cur­rent brain, and con­clude: ‘Even if I thought killing were right, killing would still be wrong.’ But this doesn’t show that killing-is-wrong is a fixed fact from out­side you.”

Obert: “Like, say, 2 + 3 = 5 is a fixed fact. Eliezer wrote: ‘If some­thing ap­pears to be the same re­gard­less of what any­one thinks, then maybe that’s be­cause it ac­tu­ally is the same re­gard­less of what any­one thinks.’ I’d say that sub­text is pretty clear!”

Subhan: “On the con­trary. Naively, you might ima­gine your fu­ture self think­ing dif­fer­ently of a thing, and visu­al­ize that the thing wouldn’t thereby change, and con­clude that the thing ex­is­ted out­side you. Eliezer shows how this is not ne­ces­sar­ily the case. So you shouldn’t trust your in­tu­ition that the thing is ob­ject­ive—it might be that the thing ex­ists out­side you, or it might not. It has to be ar­gued sep­ar­ately from the feel­ing of sub­junct­ive ob­jectiv­ity. In the case of 2 + 3 = 5, it’s at least reas­on­able to won­der if math ex­is­ted be­fore hu­mans. Phys­ics it­self seems to be made of math, and if we don’t tell a story where phys­ics was around be­fore hu­mans could ob­serve it, it’s hard to give a co­her­ent ac­count of how we got here. But there’s not the slight­est evid­ence that mor­al­ity was at work in the uni­verse be­fore hu­mans got here. We cre­ated it.”

Obert: “I know some very wise chil­dren who would dis­agree with you.”

Subhan: “Then they’re wrong! If chil­dren learned in school that it was okay to steal, they would grow up be­liev­ing it was okay to steal.”

Obert: “Not if they saw that steal­ing hurt the other per­son, and felt em­pathy for their pain. Em­pathy is a hu­man uni­ver­sal.”

Subhan: “So we take a step back and say that evol­u­tion cre­ated the emo­tions that gave rise to mor­al­ity, it doesn’t put mor­al­ity any­where out­side us. But what you say might not even be true—if theft weren’t con­sidered a crime, the other child might not feel so hurt by it. And re­gard­less, it is rare to find any child cap­able of fully re­con­sid­er­ing the moral teach­ings of its so­ci­ety.”

Obert: “I hear that, in a re­mark­able sim­il­ar­ity to Eliezer, your par­ents were Ortho­dox Jew­ish and you broke with re­li­gion as a very young child.”

Subhan: “I doubt that I was in­tern­ally gen­er­at­ing de novo moral philo­sophy. I was prob­ably just wield­ing, against Juda­ism, the mor­al­ity of the sci­ence fic­tion that ac­tu­ally so­cial­ized me.”

Obert: “Per­haps you un­der­es­tim­ate your­self. How much sci­ence fic­tion had you read at the age of five, when you real­ized it was dumb to re­cite Hebrew pray­ers you couldn’t un­der­stand? Chil­dren may see er­rors that adults are too ad­ept at fool­ing them­selves to real­ize.”

Subhan: “Hah! In all prob­ab­il­ity, if the teacher had in fact said that it was okay to take things from other chil­dren’s back­packs, the chil­dren would in fact have thought it was right to steal.”

Obert: “Even if true, that doesn’t prove any­thing. It is quite co­her­ent to sim­ul­tan­eously hold that:

  • “Steal­ing is wrong.”

  • “If a neut­rino storm makes me be­lieve ‘steal­ing is right’, then steal­ing is wrong.”

  • “If a neut­rino storm makes me be­lieve ‘steal­ing is right’, then I will say, ‘If a neut­rino storm makes me be­lieve ″steal­ing is wrong″, then steal­ing is right.’”

Subhan: “Fine, it’s co­her­ent, but that doesn’t mean it’s true. The mor­al­ity that the child has in fact learned from the teacher—or their par­ents, or the other chil­dren, or the tele­vi­sion, or their par­ents’ sci­ence fic­tion col­lec­tion—doesn’t say, ‘Don’t steal be­cause the teacher says so.’ The learned mor­al­ity just says, ‘Don’t steal.’ The cog­nit­ive pro­ced­ure by which the chil­dren were taught to judge, does not have an in­ternal de­pend­ency on what the chil­dren be­lieve the teacher be­lieves. That’s why, in their moral ima­gin­a­tion, it feels ob­ject­ive. But where did they ac­quire that mor­al­ity in the first place? From the teacher!”

Obert: “So? I don’t un­der­stand—you’re say­ing that be­cause they learned about mor­al­ity from the teacher, they should think that mor­al­ity has to be about the teacher? That they should think the teacher has the power to make it right to steal? How does that fol­low? It is quite co­her­ent to sim­ul­tan­eously hold that—”

Subhan: “I’m say­ing that they got the mor­al­ity from the teacher! Not from some mys­ter­i­ous light in the sky!”

Obert: “Look, I too read sci­ence fic­tion and fantasy as a child, and I think I may have been to some de­gree so­cial­ized by it—”

Subhan: “What a re­mark­able co­in­cid­ence.”

Obert: “The stor­ies taught me that it was right to care about people who were dif­fer­ent from me—ali­ens with strange shapes, ali­ens made of some­thing other than car­bon atoms, AIs who had been cre­ated rather than evolved, even things that didn’t think like a hu­man. But none of the stor­ies ever said, ‘You should care about people of dif­fer­ent shapes and sub­strates be­cause sci­ence fic­tion told you to do it, and what sci­ence fic­tion says, goes.’ I wouldn’t have bought that.”

Subhan: “Are you sure you wouldn’t have? That’s how re­li­gion works.”

Obert: “Didn’t work on you. Any­way, the nov­els said to care about the ali­ens be­cause they had in­ner lives and joys—or be­cause I wouldn’t want ali­ens to mis­treat hu­mans—or be­cause shape and sub­strate never had any­thing to do with what makes a per­son a per­son. And you know, that still seems to me like a good jus­ti­fic­a­tion.”

Subhan: “Of course; you were told it was a good jus­ti­fic­a­tion—maybe not dir­ectly, but the au­thor showed other char­ac­ters re­spond­ing to the ar­gu­ment.”

Obert: “It’s not like the sci­ence fic­tion writers were mak­ing up their mor­al­ity from scratch. They were work­ing at the end of a chain of moral ar­gu­ments and de­bates that stretches back to the Greeks, prob­ably to be­fore writ­ing, maybe to be­fore the dawn of mod­ern hu­man­ity. You can learn mor­al­ity, not just get pressed into it like a Jello mold. If you learn 2 + 3 = 5 from a teacher, it doesn’t mean the teacher has the power to add two sheep to three sheep and get six sheep. If you would have spouted back ‘2 + 3 = 6’ if the teacher said so, that doesn’t change the sheep, it just means that you don’t really un­der­stand the sub­ject. So too with mor­al­ity.”

Subhan: “Okay, let me try a dif­fer­ent tack. You, I take it, agree with both of these state­ments:”

  • “If I pre­ferred to kill people, it would not be­come right to kill people.”

  • “If I pre­ferred to eat an­chovy piz­zas, it would be­come right to eat an­chovy piz­zas.”

Obert: “Well, there are vari­ous caveats I’d at­tach to both of those. Like, in any cir­cum­stance where I really did prefer to kill someone, there’d be a high prob­ab­il­ity he was about to shoot me, or some­thing. And there’s all kinds of ways that eat­ing an an­chovy pizza could be wrong, like if I was already over­weight. And I don’t claim to be cer­tain of any­thing when it comes to mor­al­ity. But on the whole, and omit­ting all ob­jec­tions and knock-on ef­fects, I agree.”

Subhan: “It’s that second state­ment I’m really in­ter­ested in. How does your want­ing to eat an an­chovy pizza make it right?”

Obert: “Be­cause ceteris paribus, in the course of or­din­ary life as we know it, and bar­ring un­spe­cified side ef­fects, it is good for sen­tient be­ings to get what they want.”

Subhan: “And why doesn’t that ap­ply to the bit about killing, then?”

Obert: “Be­cause the other per­son doesn’t want to die. Look, the whole reason why it’s right in the first place for me to eat pep­p­er­oni pizza—the ori­ginal jus­ti­fic­a­tion—is that I en­joy do­ing so. Eat­ing pep­p­er­oni pizza makes me happy, which is ceteris paribus a good thing. And eat­ing an­chovy pizza—blegh! Ce­teris paribus, it’s not good for sen­tient be­ings to ex­per­i­ence dis­gust­ing tastes. But if my taste in pizza changes, that changes the con­sequn­e­ces of eat­ing, which changes the moral jus­ti­fic­a­tion, and so the moral judg­ment changes as well. But the reas­ons for not killing are in terms of the other per­son hav­ing an in­ner life that gets snuffed out—a fact that doesn’t change de­pend­ing on my own state of mind.”

Subhan: “Oh? I was guess­ing that the dif­fer­ence had some­thing to do with the so­cial dis­ap­proval that would be leveled at murder, but not at eat­ing an­chovy pizza.”

Obert: “As usual, your awk­ward at­tempts at ra­tion­al­ism have put you out of touch with self-evid­ent moral truths. That’s just not how I, or other real people, ac­tu­ally think! If I want to bleep bleep bleep a con­sent­ing adult, it doesn’t mat­ter whether so­ci­ety ap­proves. So­ci­ety can go bleep bleep bleep bleep bleep

Subhan: “Or so sci­ence fic­tion taught you.”

Obert: “Spider Robin­son’s sci­ence fic­tion, to be pre­cise. ‘Whatever turns you on’ shall be the whole of the law. So long as the ‘you’ is plural.”

Subhan: “So that’s where you got that par­tic­u­lar self-evid­ent moral truth. Was it also Spider Robin­son who told you that it was self-evid­ent?”

Obert: “No, I thought about that for a while, and then de­cided my­self.”

Subhan: “You seem to be pay­ing re­mark­ably close at­ten­tion to what people want. Yet you in­sist that what val­id­ates this at­ten­tion, is some ex­ternal stand­ard that makes the sat­is­fac­tion of de­sires, good. Can’t you just ad­mit that, by em­pathy and vi­cari­ous ex­per­i­ence and evolved fel­low-feel­ing, you want oth­ers to get what they want? When does this ex­ternal stand­ard ever say that it’s good for some­thing to hap­pen that someone doesn’t want?”

Obert: “Every time you’ve got to tell your child to lay off the ice cream, he’ll grow more fat cells that will make it im­possible for him to lose weight as an adult.”

Subhan: “And could some­thing good hap­pen that no one wanted?”

Obert: “I rather ex­pect so. I don’t think we’re all en­tirely past our child­hoods. In some ways the hu­man spe­cies it­self strikes me as be­ing a sort of tod­dler in the ‘No!’ stage.”

Subhan: “Look, there’s a per­fectly nor­mal and non-mys­ter­i­ous chain of caus­al­ity that de­scribes where mor­al­ity comes from, and it’s not from out­side hu­mans. If you’d been told that killing was right, or if you’d evolved to en­joy killing—much more than we already do, I mean—or if you really did have a mini-stroke that dam­aged your frontal lobe, then you’d be go­ing around say­ing, ‘Killing is right re­gard­less of what any­one thinks of it’. No great light in the sky would cor­rect you. There is noth­ing else to the story.”

Obert: “Really, I think that in this whole de­bate between us, there is sur­pris­ingly litle in­form­a­tion to be gained by such ob­ser­va­tions as ‘You only say that be­cause your brain makes you say it.’ If a neut­rino storm hit me, I might say ‘2 + 3 = 6’, but that wouldn’t change arith­metic. It would just make my brain com­pute some­thing other than arith­metic. And these vari­ous mis­for­tunes that you’ve de­scribed, wouldn’t change the crime of murder. They would just make my brain com­pute some­thing other than mor­al­ity.”

Part of The Metaethics Sequence

Next post: “Changing Your Metaethics

Pre­vi­ous post: “Math is Sub­junct­ively Ob­ject­ive