Could Anything Be Right?

Years ago, Eliezer1999 was con­vinced that he knew noth­ing about moral­ity.

For all he knew, moral­ity could re­quire the ex­ter­mi­na­tion of the hu­man species; and if so he saw no virtue in tak­ing a stand against moral­ity, be­cause he thought that, by defi­ni­tion, if he pos­tu­lated that moral fact, that meant hu­man ex­tinc­tion was what “should” be done.

I thought I could figure out what was right, per­haps, given enough rea­son­ing time and enough facts, but that I cur­rently had no in­for­ma­tion about it. I could not trust evolu­tion which had built me. What foun­da­tion did that leave on which to stand?

Well, in­deed Eliezer1999 was mas­sively mis­taken about the na­ture of moral­ity, so far as his ex­plic­itly rep­re­sented philos­o­phy went.

But as David­son once ob­served, if you be­lieve that “beavers” live in deserts, are pure white in color, and weigh 300 pounds when adult, then you do not have any be­liefs about beavers, true or false. You must get at least some of your be­liefs right, be­fore the re­main­ing ones can be wrong about any­thing.

My be­lief that I had no in­for­ma­tion about moral­ity was not in­ter­nally con­sis­tent.

Say­ing that I knew noth­ing felt vir­tu­ous, for I had once been taught that it was vir­tu­ous to con­fess my ig­no­rance. “The only thing I know is that I know noth­ing,” and all that. But in this case I would have been bet­ter off con­sid­er­ing the ad­mit­tedly ex­ag­ger­ated say­ing, “The great­est fool is the one who is not aware they are wise.” (This is nowhere near the great­est kind of fool­ish­ness, but it is a kind of fool­ish­ness.)

Was it wrong to kill peo­ple? Well, I thought so, but I wasn’t sure; maybe it was right to kill peo­ple, though that seemed less likely.

What kind of pro­ce­dure would an­swer whether it was right to kill peo­ple? I didn’t know that ei­ther, but I thought that if you built a generic su­per­in­tel­li­gence (what I would later la­bel a “ghost of perfect empti­ness”) then it could, you know, rea­son about what was likely to be right and wrong; and since it was su­per­in­tel­li­gent, it was bound to come up with the right an­swer.

The prob­lem that I some­how man­aged not to think too hard about, was where the su­per­in­tel­li­gence would get the pro­ce­dure that dis­cov­ered the pro­ce­dure that dis­cov­ered the pro­ce­dure that dis­cov­ered moral­ity—if I couldn’t write it into the start state that wrote the suc­ces­sor AI that wrote the suc­ces­sor AI.

As Mar­cello Her­reshoff later put it, “We never bother run­ning a com­puter pro­gram un­less we don’t know the out­put and we know an im­por­tant fact about the out­put.” If I knew noth­ing about moral­ity, and did not even claim to know the na­ture of moral­ity, then how could I con­struct any com­puter pro­gram what­so­ever—even a “su­per­in­tel­li­gent” one or a “self-im­prov­ing” one—and claim that it would out­put some­thing called “moral­ity”?

There are no-free-lunch the­o­rems in com­puter sci­ence—in a max­en­tropy uni­verse, no plan is bet­ter on av­er­age than any other. If you have no knowl­edge at all about “moral­ity”, there’s also no com­pu­ta­tional pro­ce­dure that will seem more likely than oth­ers to com­pute “moral­ity”, and no meta-pro­ce­dure that’s more likely than oth­ers to pro­duce a pro­ce­dure that com­putes “moral­ity”.

I thought that surely even a ghost of perfect empti­ness, find­ing that it knew noth­ing of moral­ity, would see a moral im­per­a­tive to think about moral­ity.

But the difficulty lies in the word think. Think­ing is not an ac­tivity that a ghost of perfect empti­ness is au­to­mat­i­cally able to carry out. Think­ing re­quires run­ning some spe­cific com­pu­ta­tion that is the thought. For a re­flec­tive AI to de­cide to think, re­quires that it know some com­pu­ta­tion which it be­lieves is more likely to tell it what it wants to know, than con­sult­ing an Ouija board; the AI must also have a no­tion of how to in­ter­pret the out­put.

If one knows noth­ing about moral­ity, what does the word “should” mean, at all? If you don’t know whether death is right or wrong—and don’t know how you can dis­cover whether death is right or wrong—and don’t know whether any given pro­ce­dure might out­put the pro­ce­dure for say­ing whether death is right or wrong—then what do these words, “right” and “wrong”, even mean?

If the words “right” and “wrong” have noth­ing baked into them—no start­ing point—if ev­ery­thing about moral­ity is up for grabs, not just the con­tent but the struc­ture and the start­ing point and the de­ter­mi­na­tion pro­ce­dure—then what is their mean­ing? What dis­t­in­guishes, “I don’t know what is right” from “I don’t know what is wakalixes”?

A sci­en­tist may say that ev­ery­thing is up for grabs in sci­ence, since any the­ory may be dis­proven; but then they have some idea of what would count as ev­i­dence that could dis­prove the the­ory. Could there be some­thing that would change what a sci­en­tist re­garded as ev­i­dence?

Well, yes, in fact; a sci­en­tist who read some Karl Pop­per and thought they knew what “ev­i­dence” meant, could be pre­sented with the co­her­ence and unique­ness proofs un­der­ly­ing Bayesian prob­a­bil­ity, and that might change their defi­ni­tion of ev­i­dence. They might not have had any ex­plicit no­tion, in ad­vance, that such a proof could ex­ist. But they would have had an im­plicit no­tion. It would have been baked into their brains, if not ex­plic­itly rep­re­sented therein, that such-and-such an ar­gu­ment would in fact per­suade them that Bayesian prob­a­bil­ity gave a bet­ter defi­ni­tion of “ev­i­dence” than the one they had been us­ing.

In the same way, you could say, “I don’t know what moral­ity is, but I’ll know it when I see it,” and make sense.

But then you are not re­bel­ling com­pletely against your own evolved na­ture. You are sup­pos­ing that what­ever has been baked into you to rec­og­nize “moral­ity”, is, if not ab­solutely trust­wor­thy, then at least your ini­tial con­di­tion with which you start de­bat­ing. Can you trust your moral in­tu­itions to give you any in­for­ma­tion about moral­ity at all, when they are the product of mere evolu­tion?

But if you dis­card ev­ery pro­ce­dure that evolu­tion gave you and all its prod­ucts, then you dis­card your whole brain. You dis­card ev­ery­thing that could po­ten­tially rec­og­nize moral­ity when it sees it. You dis­card ev­ery­thing that could po­ten­tially re­spond to moral ar­gu­ments by up­dat­ing your moral­ity. You even un­wind past the un­winder: you dis­card the in­tu­itions un­der­ly­ing your con­clu­sion that you can’t trust evolu­tion to be moral. It is your ex­ist­ing moral in­tu­itions that tell you that evolu­tion doesn’t seem like a very good source of moral­ity. What, then, will the words “right” and “should” and “bet­ter” even mean?

Hu­mans do not perfectly rec­og­nize truth when they see it, and hunter-gath­er­ers do not have an ex­plicit con­cept of the Bayesian crite­rion of ev­i­dence. But all our sci­ence and all our prob­a­bil­ity the­ory was built on top of a chain of ap­peals to our in­stinc­tive no­tion of “truth”. Had this core been flawed, there would have been noth­ing we could do in prin­ci­ple to ar­rive at the pre­sent no­tion of sci­ence; the no­tion of sci­ence would have just sounded com­pletely un­ap­peal­ing and pointless.

One of the ar­gu­ments that might have shaken my teenage self out of his mis­take, if I could have gone back in time to ar­gue with him, was the ques­tion:

Could there be some moral­ity, some given right­ness or wrong­ness, that hu­man be­ings do not per­ceive, do not want to per­ceive, will not see any ap­peal­ing moral ar­gu­ment for adopt­ing, nor any moral ar­gu­ment for adopt­ing a pro­ce­dure that adopts it, etcetera? Could there be a moral­ity, and our­selves ut­terly out­side its frame of refer­ence? But then what makes this thing moral­ity—rather than a stone tablet some­where with the words ‘Thou shalt mur­der’ writ­ten on them, with ab­solutely no jus­tifi­ca­tion offered?

So all this sug­gests that you should be will­ing to ac­cept that you might know a lit­tle about moral­ity. Noth­ing un­ques­tion­able, per­haps, but an ini­tial state with which to start ques­tion­ing your­self. Baked into your brain but not ex­plic­itly known to you, per­haps; but still, that which your brain would rec­og­nize as right is what you are talk­ing about. You will ac­cept at least enough of the way you re­spond to moral ar­gu­ments as a start­ing point, to iden­tify “moral­ity” as some­thing to think about.

But that’s a rather large step.

It im­plies ac­cept­ing your own mind as iden­ti­fy­ing a moral frame of refer­ence, rather than all moral­ity be­ing a great light shin­ing from be­yond (that in prin­ci­ple you might not be able to per­ceive at all). It im­plies ac­cept­ing that even if there were a light and your brain de­cided to rec­og­nize it as “moral­ity”, it would still be your own brain that rec­og­nized it, and you would not have evaded causal re­spon­si­bil­ity—or evaded moral re­spon­si­bil­ity ei­ther, on my view.

It im­plies drop­ping the no­tion that a ghost of perfect empti­ness will nec­es­sar­ily agree with you, be­cause the ghost might oc­cupy a differ­ent moral frame of refer­ence, re­spond to differ­ent ar­gu­ments, be ask­ing a differ­ent ques­tion when it com­putes what-to-do-next.

And if you’re will­ing to bake at least a few things into the very mean­ing of this topic of “moral­ity”, this qual­ity of right­ness that you are talk­ing about when you talk about “right­ness”—if you’re will­ing to ac­cept even that moral­ity is what you ar­gue about when you ar­gue about “moral­ity”—then why not ac­cept other in­tu­itions, other pieces of your­self, into the start­ing point as well?

Why not ac­cept that, ce­teris paribus, joy is prefer­able to sor­row?

You might later find some ground within your­self or built upon your­self with which to crit­i­cize this—but why not ac­cept it for now? Not just as a per­sonal prefer­ence, mind you; but as some­thing baked into the ques­tion you ask when you ask “What is truly right”?

But then you might find that you know rather a lot about moral­ity! Noth­ing cer­tain—noth­ing un­ques­tion­able—noth­ing unar­guable—but still, quite a bit of in­for­ma­tion. Are you will­ing to re­lin­quish your Socratean ig­no­rance?

I don’t ar­gue by defi­ni­tions, of course. But if you claim to know noth­ing at all about moral­ity, then you will have prob­lems with the mean­ing of your words, not just their plau­si­bil­ity.