Moral Error and Moral Disagreement

Fol­lowup to: Insep­a­rably Right, Sort­ing Peb­bles Into Cor­rect Heaps

Richard Chap­pell, a pro, writes:

“When Bob says “Abor­tion is wrong”, and Sally says, “No it isn’t”, they are dis­agree­ing with each other.

I don’t see how Eliezer can ac­com­mo­date this. On his ac­count, what Bob as­serted is true iff abor­tion is pro­hibited by the moral­ity_Bob norms. How can Sally dis­agree? There’s no dis­put­ing (we may sup­pose) that abor­tion is in­deed pro­hibited by moral­ity_Bob...

Since there is moral dis­agree­ment, what­ever Eliezer pur­ports to be analysing here, it is not moral­ity.”

The phe­nom­ena of moral dis­agree­ment, moral er­ror, and moral progress, on ter­mi­nal val­ues, are the pri­mary drivers be­hind my metaethics. Think of how sim­ple Friendly AI would be if there were no moral dis­agree­ments, moral er­rors, or moral progress!

Richard claims, “There’s no dis­put­ing (we may sup­pose) that abor­tion is in­deed pro­hibited by moral­ity_Bob.”

We may not sup­pose, and there is dis­put­ing. Bob does not have di­rect, un­medi­ated, veridi­cal ac­cess to the out­put of his own moral­ity.

I tried to de­scribe moral­ity as a “com­pu­ta­tion”. In ret­ro­spect, I don’t think this is func­tion­ing as the Word of Power that I thought I was emit­ting.

Let us read, for “com­pu­ta­tion”, “ideal­ized ab­stract dy­namic”—maybe that will be a more com­fortable la­bel to ap­ply to moral­ity.

Even so, I would have thought it ob­vi­ous that com­pu­ta­tions may be the sub­jects of mys­tery and er­ror. Maybe it’s not as ob­vi­ous out­side com­puter sci­ence?

Disagree­ment has two pre­req­ui­sites: the pos­si­bil­ity of agree­ment and the pos­si­bil­ity of er­ror. For two peo­ple to agree on some­thing, there must be some­thing they are agree­ing about, a refer­ent held in com­mon. And it must be pos­si­ble for an “er­ror” to take place, a con­flict be­tween “P” in the map and not-P in the ter­ri­tory. Where these two pre­req­ui­sites are pre­sent, Sally can say to Bob: “That thing we were just both talk­ing about—you are in er­ror about it.”

Richard’s ob­jec­tion would seem in the first place to rule out the pos­si­bil­ity of moral er­ror, from which he de­rives the im­pos­si­bil­ity of moral agree­ment.

So: does my metaethics rule out moral er­ror? Is there no dis­put­ing that abor­tion is in­deed pro­hibited by moral­ity_Bob?

This is such a strange idea that I find my­self won­der­ing what the heck Richard could be think­ing. My best guess is that Richard, per­haps hav­ing not read all the posts in this se­quence, is tak­ing my no­tion of moral­ity_Bob to re­fer to a flat, static list of val­u­a­tions ex­plic­itly as­serted by Bob. “Abor­tion is wrong” would be on Bob’s list, and there would be no dis­put­ing that.

But on the con­trary, I con­ceive of moral­ity_Bob as some­thing that un­folds into Bob’s moral­ity—like the way one can de­scribe in 6 states and 2 sym­bols a Tur­ing ma­chine that will write 4.640 × 101439 1s to its tape be­fore halt­ing.

So moral­ity_Bob refers to a com­pact folded speci­fi­ca­tion, and not a flat list of out­puts. But still, how could Bob be wrong about the out­put of his own moral­ity?

In man­i­fold ob­vi­ous and non-ob­vi­ous ways:

Bob could be em­piri­cally mis­taken about the state of fe­tuses, per­haps be­liev­ing fe­tuses to be aware of the out­side world. (Cor­rect­ing this might change Bob’s in­stru­men­tal val­ues but not ter­mi­nal val­ues.)

Bob could have formed his be­liefs about what con­sti­tuted “per­son­hood” in the pres­ence of con­fu­sion about the na­ture of con­scious­ness, so that if Bob were fully in­formed about con­scious­ness, Bob would not have been tempted to talk about “the be­gin­ning of life” or “the hu­man kind” in or­der to define per­son­hood. (This changes Bob’s ex­pressed ter­mi­nal val­ues; af­ter­ward he will state differ­ent gen­eral rules about what sort of phys­i­cal things are ends in them­selves.)

So those are the ob­vi­ous moral er­rors—in­stru­men­tal er­rors driven by em­piri­cal mis­takes; and er­ro­neous gen­er­al­iza­tions about ter­mi­nal val­ues, driven by failure to con­sider moral ar­gu­ments that are valid but hard to find in the search space.

Then there are less ob­vi­ous sources of moral er­ror: Bob could have a list of mind-in­fluenc­ing con­sid­er­a­tions that he con­sid­ers morally valid, and a list of other mind-in­fluenc­ing con­sid­er­a­tions that Bob con­sid­ers morally in­valid. Maybe Bob was raised a Chris­tian and now con­sid­ers that cul­tural in­fluence to be in­valid. But, un­known to Bob, when he weighs up his val­ues for and against abor­tion, the in­fluence of his Chris­tian up­bring­ing comes in and dis­torts his sum­ming of value-weights. So Bob be­lieves that the out­put of his cur­rent val­i­dated moral be­liefs is to pro­hibit abor­tion, but ac­tu­ally this is a lef­tover of his child­hood and not the out­put of those be­liefs at all.

(Note that Robin Han­son and I seem to dis­agree, in a case like this, as to ex­actly what de­gree we should take Bob’s word about what his morals are.)

Or Bob could be­lieve that the word of God de­ter­mines moral truth and that God has pro­hibited abor­tion in the Bible. Then Bob is mak­ing metaeth­i­cal mis­takes, caus­ing his mind to malfunc­tion in a highly gen­eral way, and add moral gen­er­al­iza­tions to his be­lief pool, which he would not do if veridi­cal knowl­edge of the uni­verse de­stroyed his cur­rent and in­co­her­ent metaethics.

Now let us turn to the dis­agree­ment be­tween Sally and Bob.

You could sug­gest that Sally is say­ing to Bob, “Abor­tion is al­lowed by moral­ity_Bob”, but that seems a bit over­sim­plified; it is not psy­cholog­i­cally or morally re­al­is­tic.

If Sally and Bob were un­re­al­is­ti­cally so­phis­ti­cated, they might de­scribe their dis­pute as fol­lows:

Bob: “Abor­tion is wrong.”

Sally: “Do you think that this is some­thing of which most hu­mans ought to be per­suad­able?”

Bob: “Yes, I do. Do you think abor­tion is right?”

Sally: “Yes, I do. And I don’t think that’s be­cause I’m a psy­chopath by com­mon hu­man stan­dards. I think most hu­mans would come to agree with me, if they knew the facts I knew, and heard the same moral ar­gu­ments I’ve heard.”

Bob: “I think, then, that we must have a moral dis­agree­ment: since we both be­lieve our­selves to be a shared moral frame of refer­ence on this is­sue, and yet our moral in­tu­itions say differ­ent things to us.”

Sally: “Well, it is not log­i­cally nec­es­sary that we have a gen­uine dis­agree­ment. We might be mis­taken in be­liev­ing our­selves to mean the same thing by the words right and wrong, since nei­ther of us can in­tro­spec­tively re­port our own moral refer­ence frames or un­fold them fully.”

Bob: “But if the mean­ing is similar up to the third dec­i­mal place, or suffi­ciently similar in some re­spects that it ought to be de­liv­er­ing similar an­swers on this par­tic­u­lar is­sue, then, even if our moral­ities are not in-prin­ci­ple iden­ti­cal, I would not hes­i­tate to in­voke the in­tu­itions for transper­sonal moral­ity.”

Sally: “I agree. Un­til proven oth­er­wise, I am in­clined to talk about this ques­tion as if it is the same ques­tion unto us.”

Bob: “So I say ‘Abor­tion is wrong’ with­out fur­ther qual­ifi­ca­tion or spe­cial­iza­tion on what wrong means unto me.”

Sally: “And I think that abor­tion is right. We have a dis­agree­ment, then, and at least one of us must be mis­taken.”

Bob: “Un­less we’re ac­tu­ally choos­ing differ­ently be­cause of in-prin­ci­ple un­re­solv­able differ­ences in our moral frame of refer­ence, as if one of us were a pa­per­clip max­i­mizer. In that case, we would be mu­tu­ally mis­taken in our be­lief that when we talk about do­ing what is right, we mean the same thing by right. We would agree that we have a dis­agree­ment, but we would both be wrong.”

Now, this is not ex­actly what most peo­ple are ex­plic­itly think­ing when they en­gage in a moral dis­pute—but it is how I would cash out and nat­u­ral­ize their in­tu­itions about transper­sonal moral­ity.

Richard also says, “Since there is moral dis­agree­ment...” This seems like a prime case of what I call naive philo­soph­i­cal re­al­ism—the be­lief that philo­soph­i­cal in­tu­itions are di­rect un­medi­ated veridi­cal pass­ports to philo­soph­i­cal truth.

It so hap­pens that I agree that there is such a thing as moral dis­agree­ment. To­mor­row I will en­deavor to jus­tify, in ful­ler de­tail, how this state­ment can pos­si­bly make sense in a re­duc­tion­is­tic nat­u­ral uni­verse. So I am not dis­put­ing this par­tic­u­lar propo­si­tion. But I note, in pass­ing, that Richard can­not jus­tifi­ably as­sert the ex­is­tence of moral dis­agree­ment as an ir­refutable premise for dis­cus­sion, though he could con­sider it as an ap­par­ent da­tum. You can­not take as ir­refutable premises, things that you have not ex­plained ex­actly; for then what is it that is cer­tain to be true?

I can­not help but note the re­sem­blance to Richard’s as­sump­tion that “there’s no dis­put­ing” that abor­tion is in­deed pro­hibited by moral­ity_Bob—the as­sump­tion that Bob has di­rect veridi­cal un­medi­ated ac­cess to the fi­nal un­folded out­put of his own moral­ity.

Per­haps Richard means that we could sup­pose that abor­tion is in­deed pro­hibited by moral­ity_Bob, and al­lowed by moral­ity_Sally, there be­ing at least two pos­si­ble minds for whom this would be true. Then the two minds might be mis­taken about be­liev­ing them­selves to dis­agree. Ac­tu­ally they would sim­ply be di­rected by differ­ent al­gorithms.

You can­not have a dis­agree­ment about which al­gorithm should di­rect your ac­tions, with­out first hav­ing the same mean­ing of should—and no mat­ter how you try to phrase this in terms of “what ought to di­rect your ac­tions” or “right ac­tions” or “cor­rect heaps of peb­bles”, in the end you will be left with the em­piri­cal fact that it is pos­si­ble to con­struct minds di­rected by any co­her­ent util­ity func­tion.

When a pa­per­clip max­i­mizer and a pen­cil max­i­mizer do differ­ent things, they are not dis­agree­ing about any­thing, they are just differ­ent op­ti­miza­tion pro­cesses. You can­not de­tach should-ness from any spe­cific crite­rion of should-ness and be left with a pure empty should-ness that the pa­per­clip max­i­mizer and pen­cil max­i­mizer can be said to dis­agree about—un­less you cover “dis­agree­ment” to in­clude differ­ences where two agents have noth­ing to say to each other.

But this would be an ex­treme po­si­tion to take with re­spect to your fel­low hu­mans, and I recom­mend against do­ing so. Even a psy­chopath would still be in a com­mon moral refer­ence frame with you, if, fully in­formed, they would de­cide to take a pill that would make them non-psy­chopaths. If you told me that my abil­ity to care about other peo­ple was neu­rolog­i­cally dam­aged, and you offered me a pill to fix it, I would take it. Now, per­haps some psy­chopaths would not be per­suad­able in-prin­ci­ple to take the pill that would, by our stan­dards, “fix” them. But I note the pos­si­bil­ity to em­pha­size what an ex­treme state­ment it is to say of some­one:

“We have noth­ing to ar­gue about, we are only differ­ent op­ti­miza­tion pro­cesses.”

That should be re­served for pa­per­clip max­i­miz­ers, not used against hu­mans whose ar­gu­ments you don’t like.

Part of The Me­taethics Sequence

Next post: “Ab­stracted Ideal­ized Dy­nam­ics

Pre­vi­ous post: “Sort­ing Peb­bles Into Cor­rect Heaps