“Moral” as a preference label

Note: work­ing on a re­search agenda, hence the large amount of small in­di­vi­d­ual posts, to have things to link to in the main doc­u­ments.

In my quest to syn­the­sise hu­man prefer­ences, I’ve oc­ca­sion­ally been asked whether I dis­t­in­guish moral prefer­ences from other types of prefer­ences—for ex­am­ple, whether prefer­ences for Abba or Beethoven, or av­o­cado or sausages, should rank as high as hu­man rights or free­dom of speech.

The an­swer is, of course not. But these are not the sort of things that should be built into the sys­tem by hand. This should be re­flected in the meta-prefer­ences. We la­bel cer­tain prefer­ences “moral”, and we of­ten have the be­lief that these should have pri­or­ity, to some ex­tent, over merely “self­ish” prefer­ences (the ex­tent of this be­lief varies from per­son to per­son, of course).

I de­liber­ately wrote the wrong word there for this for­mal­ism—we don’t have the “be­lief” that moral prefer­ences are more im­por­tant, we have the meta-prefer­ence that a cer­tain class of be­liefs, la­bel­led “moral”, what­ever that turns out to mean, should be given greater weight. This is es­pe­cially the case as there are a lot of cases where it is very un­clear if a prefer­ence is moral or not (many peo­ple have strong moral-ish prefer­ences over main­stream cul­tural and en­ter­tain­ment choices).

This is an ex­am­ple of the sort of challenges that a prefer­ence syn­the­sis pro­cess should be able to figure out on its own. If the method needs to be con­stantly tweaked to get over ev­ery small prob­lem of defi­ni­tion, then it can­not work. As always, how­ever, it need not get ev­ery­thing ex­actly right; in­deed, it needs to be ro­bust enough that it doesn’t change much if a bor­der­line meta-prefer­ence such as “ev­ery­one should know their own his­tory” gets la­bel­led as moral or not.