Does Your Morality Care What You Think?

Fol­lowup to: Math is Sub­junc­tively Ob­jec­tive, The Mo­ral Void, Is Mo­ral­ity Given?

Thus I re­call the study, though I can­not re­call the cita­tion:

Chil­dren, at some rel­a­tively young age, were found to dis­t­in­guish be­tween:

  • The teacher, by say­ing that we’re al­lowed to stand on our desks, can make it right to do so.

  • The teacher, by say­ing that I’m al­lowed to take some­thing from an­other child’s back­pack, can­not make it right to do so.

Obert: “Well, I don’t know the cita­tion, but it sounds like a fas­ci­nat­ing study. So even chil­dren, then, re­al­ize that moral facts are givens, be­yond the abil­ity of teach­ers or par­ents to al­ter.”

Sub­han: “You say that like it’s a good thing. Chil­dren may also think that peo­ple in Aus­tralia have to wear heavy boots from fal­ling off the other side of the Earth.”

Obert: “Call me Peter Pan, then, be­cause I never grew up on this one. Of course it doesn’t mat­ter what the teacher says. It doesn’t mat­ter what I say. It doesn’t even mat­ter what I think. Steal­ing is wrong. Do you dis­agree?

Sub­han: “You don’t see me pick­ing your pock­ets, do you? Isn’t it enough that I choose not to steal from you—do I have to pre­tend it’s the law of the uni­verse?”

Obert: “Yes, or I can’t trust your com­mit­ment.”

Sub­han: “A… re­veal­ing re­mark. But re­ally, I don’t think that this ex­per­i­men­tal re­sult seems at all con­fus­ing, in light of the re­cent dis­cus­sion of sub­junc­tive ob­jec­tivity—a dis­cus­sion in which Eliezer strongly sup­ported my po­si­tion, by the way.”

Obert: “Really? I thought Eliezer was fi­nally com­ing out in fa­vor of my po­si­tion.”

Sub­han: “Huh? How do you get that?”

Obert: “The whole sub­text of ‘Math is Sub­junc­tively Ob­jec­tive’ is that moral­ity is just like math! Sure, we com­pute moral­ity in­side our own brains—where else would we com­pute it? But just be­cause we com­pute a quan­tity in­side our own brains, doesn’t mean that what is com­puted has a de­pen­dency on our own state of mind.”

Sub­han: “I think we must have been read­ing differ­ent Over­com­ing Bias posts! The whole sub­text of ‘Math is Sub­junc­tively Ob­jec­tive’ is to ex­plain away why moral­ity seems ob­jec­tive—to show that the feel­ing of a fixed given can arise with­out any ex­ter­nal refer­ent. When you imag­ine your­self think­ing that kil­ling is right, your brain-that-imag­ines hasn’t yet been al­tered, so you carry out that moral imag­i­na­tion with your cur­rent brain, and con­clude: ‘Even if I thought kil­ling were right, kil­ling would still be wrong.’ But this doesn’t show that kil­ling-is-wrong is a fixed fact from out­side you.”

Obert: “Like, say, 2 + 3 = 5 is a fixed fact. Eliezer wrote: ‘If some­thing ap­pears to be the same re­gard­less of what any­one thinks, then maybe that’s be­cause it ac­tu­ally is the same re­gard­less of what any­one thinks.’ I’d say that sub­text is pretty clear!”

Sub­han: “On the con­trary. Naively, you might imag­ine your fu­ture self think­ing differ­ently of a thing, and vi­su­al­ize that the thing wouldn’t thereby change, and con­clude that the thing ex­isted out­side you. Eliezer shows how this is not nec­es­sar­ily the case. So you shouldn’t trust your in­tu­ition that the thing is ob­jec­tive—it might be that the thing ex­ists out­side you, or it might not. It has to be ar­gued sep­a­rately from the feel­ing of sub­junc­tive ob­jec­tivity. In the case of 2 + 3 = 5, it’s at least rea­son­able to won­der if math ex­isted be­fore hu­mans. Physics it­self seems to be made of math, and if we don’t tell a story where physics was around be­fore hu­mans could ob­serve it, it’s hard to give a co­her­ent ac­count of how we got here. But there’s not the slight­est ev­i­dence that moral­ity was at work in the uni­verse be­fore hu­mans got here. We cre­ated it.”

Obert: “I know some very wise chil­dren who would dis­agree with you.”

Sub­han: “Then they’re wrong! If chil­dren learned in school that it was okay to steal, they would grow up be­liev­ing it was okay to steal.”

Obert: “Not if they saw that steal­ing hurt the other per­son, and felt em­pa­thy for their pain. Em­pa­thy is a hu­man uni­ver­sal.”

Sub­han: “So we take a step back and say that evolu­tion cre­ated the emo­tions that gave rise to moral­ity, it doesn’t put moral­ity any­where out­side us. But what you say might not even be true—if theft weren’t con­sid­ered a crime, the other child might not feel so hurt by it. And re­gard­less, it is rare to find any child ca­pa­ble of fully re­con­sid­er­ing the moral teach­ings of its so­ciety.”

Obert: “I hear that, in a re­mark­able similar­ity to Eliezer, your par­ents were Ortho­dox Jewish and you broke with re­li­gion as a very young child.”

Sub­han: “I doubt that I was in­ter­nally gen­er­at­ing de novo moral philos­o­phy. I was prob­a­bly just wield­ing, against Ju­daism, the moral­ity of the sci­ence fic­tion that ac­tu­ally so­cial­ized me.”

Obert: “Per­haps you un­der­es­ti­mate your­self. How much sci­ence fic­tion had you read at the age of five, when you re­al­ized it was dumb to re­cite He­brew prayers you couldn’t un­der­stand? Chil­dren may see er­rors that adults are too adept at fool­ing them­selves to re­al­ize.”

Sub­han: “Hah! In all prob­a­bil­ity, if the teacher had in fact said that it was okay to take things from other chil­dren’s back­packs, the chil­dren would in fact have thought it was right to steal.”

Obert: “Even if true, that doesn’t prove any­thing. It is quite co­her­ent to si­mul­ta­neously hold that:

  • “Steal­ing is wrong.”

  • “If a neu­trino storm makes me be­lieve ‘steal­ing is right’, then steal­ing is wrong.”

  • “If a neu­trino storm makes me be­lieve ‘steal­ing is right’, then I will say, ‘If a neu­trino storm makes me be­lieve ″steal­ing is wrong″, then steal­ing is right.’”

Sub­han: “Fine, it’s co­her­ent, but that doesn’t mean it’s true. The moral­ity that the child has in fact learned from the teacher—or their par­ents, or the other chil­dren, or the tele­vi­sion, or their par­ents’ sci­ence fic­tion col­lec­tion—doesn’t say, ‘Don’t steal be­cause the teacher says so.’ The learned moral­ity just says, ‘Don’t steal.’ The cog­ni­tive pro­ce­dure by which the chil­dren were taught to judge, does not have an in­ter­nal de­pen­dency on what the chil­dren be­lieve the teacher be­lieves. That’s why, in their moral imag­i­na­tion, it feels ob­jec­tive. But where did they ac­quire that moral­ity in the first place? From the teacher!”

Obert: “So? I don’t un­der­stand—you’re say­ing that be­cause they learned about moral­ity from the teacher, they should think that moral­ity has to be about the teacher? That they should think the teacher has the power to make it right to steal? How does that fol­low? It is quite co­her­ent to si­mul­ta­neously hold that—”

Sub­han: “I’m say­ing that they got the moral­ity from the teacher! Not from some mys­te­ri­ous light in the sky!”

Obert: “Look, I too read sci­ence fic­tion and fan­tasy as a child, and I think I may have been to some de­gree so­cial­ized by it—”

Sub­han: “What a re­mark­able co­in­ci­dence.”

Obert: “The sto­ries taught me that it was right to care about peo­ple who were differ­ent from me—aliens with strange shapes, aliens made of some­thing other than car­bon atoms, AIs who had been cre­ated rather than evolved, even things that didn’t think like a hu­man. But none of the sto­ries ever said, ‘You should care about peo­ple of differ­ent shapes and sub­strates be­cause sci­ence fic­tion told you to do it, and what sci­ence fic­tion says, goes.’ I wouldn’t have bought that.”

Sub­han: “Are you sure you wouldn’t have? That’s how re­li­gion works.”

Obert: “Didn’t work on you. Any­way, the nov­els said to care about the aliens be­cause they had in­ner lives and joys—or be­cause I wouldn’t want aliens to mis­treat hu­mans—or be­cause shape and sub­strate never had any­thing to do with what makes a per­son a per­son. And you know, that still seems to me like a good jus­tifi­ca­tion.”

Sub­han: “Of course; you were told it was a good jus­tifi­ca­tion—maybe not di­rectly, but the au­thor showed other char­ac­ters re­spond­ing to the ar­gu­ment.”

Obert: “It’s not like the sci­ence fic­tion writ­ers were mak­ing up their moral­ity from scratch. They were work­ing at the end of a chain of moral ar­gu­ments and de­bates that stretches back to the Greeks, prob­a­bly to be­fore writ­ing, maybe to be­fore the dawn of mod­ern hu­man­ity. You can learn moral­ity, not just get pressed into it like a Jello mold. If you learn 2 + 3 = 5 from a teacher, it doesn’t mean the teacher has the power to add two sheep to three sheep and get six sheep. If you would have spouted back ‘2 + 3 = 6’ if the teacher said so, that doesn’t change the sheep, it just means that you don’t re­ally un­der­stand the sub­ject. So too with moral­ity.”

Sub­han: “Okay, let me try a differ­ent tack. You, I take it, agree with both of these state­ments:”

  • “If I preferred to kill peo­ple, it would not be­come right to kill peo­ple.”

  • “If I preferred to eat an­chovy piz­zas, it would be­come right to eat an­chovy piz­zas.”

Obert: “Well, there are var­i­ous caveats I’d at­tach to both of those. Like, in any cir­cum­stance where I re­ally did pre­fer to kill some­one, there’d be a high prob­a­bil­ity he was about to shoot me, or some­thing. And there’s all kinds of ways that eat­ing an an­chovy pizza could be wrong, like if I was already over­weight. And I don’t claim to be cer­tain of any­thing when it comes to moral­ity. But on the whole, and omit­ting all ob­jec­tions and knock-on effects, I agree.”

Sub­han: “It’s that sec­ond state­ment I’m re­ally in­ter­ested in. How does your want­ing to eat an an­chovy pizza make it right?”

Obert: “Be­cause ce­teris paribus, in the course of or­di­nary life as we know it, and bar­ring un­speci­fied side effects, it is good for sen­tient be­ings to get what they want.”

Sub­han: “And why doesn’t that ap­ply to the bit about kil­ling, then?”

Obert: “Be­cause the other per­son doesn’t want to die. Look, the whole rea­son why it’s right in the first place for me to eat pep­per­oni pizza—the origi­nal jus­tifi­ca­tion—is that I en­joy do­ing so. Eat­ing pep­per­oni pizza makes me happy, which is ce­teris paribus a good thing. And eat­ing an­chovy pizza—blegh! Ce­teris paribus, it’s not good for sen­tient be­ings to ex­pe­rience dis­gust­ing tastes. But if my taste in pizza changes, that changes the con­se­quneces of eat­ing, which changes the moral jus­tifi­ca­tion, and so the moral judg­ment changes as well. But the rea­sons for not kil­ling are in terms of the other per­son hav­ing an in­ner life that gets snuffed out—a fact that doesn’t change de­pend­ing on my own state of mind.”

Sub­han: “Oh? I was guess­ing that the differ­ence had some­thing to do with the so­cial dis­ap­proval that would be lev­eled at mur­der, but not at eat­ing an­chovy pizza.”

Obert: “As usual, your awk­ward at­tempts at ra­tio­nal­ism have put you out of touch with self-ev­i­dent moral truths. That’s just not how I, or other real peo­ple, ac­tu­ally think! If I want to bleep bleep bleep a con­sent­ing adult, it doesn’t mat­ter whether so­ciety ap­proves. So­ciety can go bleep bleep bleep bleep bleep

Sub­han: “Or so sci­ence fic­tion taught you.”

Obert: “Spi­der Robin­son’s sci­ence fic­tion, to be pre­cise. ‘What­ever turns you on’ shall be the whole of the law. So long as the ‘you’ is plu­ral.”

Sub­han: “So that’s where you got that par­tic­u­lar self-ev­i­dent moral truth. Was it also Spi­der Robin­son who told you that it was self-ev­i­dent?”

Obert: “No, I thought about that for a while, and then de­cided my­self.”

Sub­han: “You seem to be pay­ing re­mark­ably close at­ten­tion to what peo­ple want. Yet you in­sist that what val­i­dates this at­ten­tion, is some ex­ter­nal stan­dard that makes the satis­fac­tion of de­sires, good. Can’t you just ad­mit that, by em­pa­thy and vi­car­i­ous ex­pe­rience and evolved fel­low-feel­ing, you want oth­ers to get what they want? When does this ex­ter­nal stan­dard ever say that it’s good for some­thing to hap­pen that some­one doesn’t want?”

Obert: “Every time you’ve got to tell your child to lay off the ice cream, he’ll grow more fat cells that will make it im­pos­si­ble for him to lose weight as an adult.”

Sub­han: “And could some­thing good hap­pen that no one wanted?”

Obert: “I rather ex­pect so. I don’t think we’re all en­tirely past our child­hoods. In some ways the hu­man species it­self strikes me as be­ing a sort of tod­dler in the ‘No!’ stage.”

Sub­han: “Look, there’s a perfectly nor­mal and non-mys­te­ri­ous chain of causal­ity that de­scribes where moral­ity comes from, and it’s not from out­side hu­mans. If you’d been told that kil­ling was right, or if you’d evolved to en­joy kil­ling—much more than we already do, I mean—or if you re­ally did have a mini-stroke that dam­aged your frontal lobe, then you’d be go­ing around say­ing, ‘Killing is right re­gard­less of what any­one thinks of it’. No great light in the sky would cor­rect you. There is noth­ing else to the story.”

Obert: “Really, I think that in this whole de­bate be­tween us, there is sur­pris­ingly litle in­for­ma­tion to be gained by such ob­ser­va­tions as ‘You only say that be­cause your brain makes you say it.’ If a neu­trino storm hit me, I might say ‘2 + 3 = 6’, but that wouldn’t change ar­ith­metic. It would just make my brain com­pute some­thing other than ar­ith­metic. And these var­i­ous mis­for­tunes that you’ve de­scribed, wouldn’t change the crime of mur­der. They would just make my brain com­pute some­thing other than moral­ity.”

Part of The Me­taethics Sequence

Next post: “Chang­ing Your Me­taethics

Pre­vi­ous post: “Math is Sub­junc­tively Ob­jec­tive