# Morality Isn’t Logical

What do I mean by “moral­ity isn’t log­i­cal”? I mean in the same sense that math­e­mat­ics is log­i­cal but liter­ary crit­i­cism isn’t: the “rea­son­ing” we use to think about moral­ity doesn’t re­sem­ble log­i­cal rea­son­ing. All sys­tems of logic, that I’m aware of, have a con­cept of proof and a method of ver­ify­ing with high de­gree of cer­tainty whether an ar­gu­ment con­sti­tutes a proof. As long as the logic is con­sis­tent (and we have good rea­son to think that many of them are), once we ver­ify a proof we can ac­cept its con­clu­sion with­out wor­ry­ing that there may be an­other proof that makes the op­po­site con­clu­sion. With moral­ity though, we have no such method, and peo­ple all the time make moral ar­gu­ments that can be re­versed or called into ques­tion by other moral ar­gu­ments. (Edit: For an ex­am­ple of this, see these posts.)

Without be­ing a sys­tem of logic, moral philo­soph­i­cal rea­son­ing likely (or at least plau­si­bly) doesn’t have any of the nice prop­er­ties that a well-con­structed sys­tem of logic would have, for ex­am­ple, con­sis­tency, val­idity, sound­ness, or even the more ba­sic prop­erty that con­sid­er­ing ar­gu­ments in a differ­ent or­der, or in a differ­ent mood, won’t cause a per­son to ac­cept an en­tirely differ­ent set of con­clu­sions. For all we know, some­body try­ing to rea­son about a moral con­cept like “fair­ness” may just be tak­ing a ran­dom walk as they move from one con­clu­sion to an­other based on moral ar­gu­ments they en­counter or think up.

In a re­cent post, Eliezer said “moral­ity is logic”, by which he seems to mean… well, I’m still not ex­actly sure what, but one in­ter­pre­ta­tion is that a per­son’s cog­ni­tion about moral­ity can be de­scribed as an al­gorithm, and that al­gorithm can be stud­ied us­ing log­i­cal rea­son­ing. (Which of course is true, but in that sense both math and liter­ary crit­i­cism as well as ev­ery other sub­ject of hu­man study would be logic.) In any case, I don’t think Eliezer is ex­plic­itly claiming that an al­gorithm-for-think­ing-about-moral­ity con­sti­tutes an al­gorithm-for-do­ing-logic, but I worry that the char­ac­ter­i­za­tion of “moral­ity is logic” may cause some con­no­ta­tions of “logic” to be in­ap­pro­pri­ately sneaked into “moral­ity”. For ex­am­ple Eliezer seems to (at least at one point) as­sume that con­sid­er­ing moral ar­gu­ments in a differ­ent or­der won’t cause a hu­man to ac­cept an en­tirely differ­ent set of con­clu­sions, and maybe this is why. To fight this po­ten­tial sneak­ing of con­no­ta­tions, I sug­gest that when you see the phrase “moral­ity is logic”, re­mind your­self that moral­ity isn’t log­i­cal.

• Ta­boo both “moral­ity” and “log­i­cal” and you may find that you and Eliezer have no dis­agree­ment.

LessWrongers rou­tinely dis­agree on what is meant by “moral­ity”. If you think “moral­ity” is am­bigu­ous, then stipu­late a mean­ing (‘moral­ity₁ is...‘) and carry on. If you think peo­ple’s dis­agree­ment about the con­tent of “moral­ity” makes it gib­ber­ish, then deny­ing that there are moral truths, or that those truths are “log­i­cal,” will equally be gib­ber­ish. Eliezer’s gen­eral prac­tice is to rea­son care­fully but in­for­mally with some­thing in the neigh­bor­hood of our col­lo­quial mean­ings of terms, when it’s clear that we could stipu­late a pre­cise defi­ni­tion that ad­e­quately ap­prox­i­mates what most peo­ple mean. Words like ‘dog’ and ‘coun­try’ and ‘num­ber’ and ‘curry’ and ‘fair­ness’ are fuzzy (if not out­right am­bigu­ous) in nat­u­ral lan­guage, but we can con­struct more rigor­ous defi­ni­tions that aren’t com­pletely se­man­ti­cally alien.

Sur­pris­ingly, we seem to be even less clear about what is meant by “logic”. A logic, sim­ply put, is a set of ex­plicit rules for gen­er­at­ing lines in a proof. And “logic,” as a hu­man prac­tice, is the use and cre­ation of such rules. But peo­ple in­for­mally speak of things as “log­i­cal” when­ever they have a ‘log­i­cal­ish vibe,’ i.e., when­ever they in­volve es­pe­cially rigor­ous ab­stract rea­son­ing.

Eliezer’s stan­dard use of ‘log­i­cal’ takes the ‘ab­stract’ part of log­i­cal­ish vibes and runs with them; he adopts the con­ven­tion that suffi­ciently care­ful purely ab­stract rea­son­ing (i.e., rea­son­ing with­out rea­son­ing about any par­tic­u­lar spa­tiotem­po­ral thing or pat­tern) is ‘log­i­cal,’ whereas rea­son­ing about con­crete things-in-the-world is ‘phys­i­cal.’ Of course, in prac­tice our rea­son­ing is usu­ally a mix of log­i­cal and phys­i­cal; but Eliezer’s con­ven­tion gives us a heuris­tic for de­ter­min­ing whether some x that we ap­peal to in rea­son­ing is log­i­cal (i.e., ab­stract, non­spa­tial) or phys­i­cal (i.e., con­crete, spa­tially lo­cated). We can eas­ily see that if the word ‘fair­ness’ de­notes any­thing (i.e., it’s not like ‘uni­corn’ or ‘square cir­cle’), it must be de­not­ing a log­i­cal/​ab­stract sort of thingie, since fair­ness isn’t some­where. (Fair­ness, un­like houses and candy, does not de­com­pose into quarks and elec­trons.)

By the same rea­son­ing, it be­comes clear that things like ‘difficulty’ and ‘the av­er­age South Afri­can male’ and ‘the set of prime num­bers’ and ‘the le­gal sys­tem of Switzer­land’ are not phys­i­cal ob­jects; there isn’t any place where difficulty liter­ally is, as though it were a Large Ob­ject hid­ing some­place just out of view. It’s an ab­strac­tion (or, in EY’s idiom, a ‘log­i­cal’ con­struct) our brains posit as a tool for think­ing, in the same fun­da­men­tal way that we posit num­bers, sets, ax­ioms, and pos­si­ble wor­lds. The posits of liter­ary the­ory are fre­quently ‘log­i­cal’ (i.e., ab­stract) in Eliezer’s sense, when they have se­man­tic can­di­dates we can stipu­late as hav­ing ad­e­quately pre­cise char­ac­ter­is­tics. Eliezer’s happy to be big-tent here, be­cause he’s do­ing do­main-gen­eral episte­mol­ogy and (meta)physics, not try­ing to lay out the pre­cise dis­tinc­tions be­tween differ­ent fields in academia. And do­ing so high­lights the im­por­tant point that rea­son­ing about what’s moral is not cat­e­gor­i­cally un­like rea­son­ing about what’s difficult, or what’s a planet, or what’s de­sir­able, or what’s com­mon, or what’s ille­gal; our nat­u­ral-lan­guage lay-us­age may un­der­de­ter­mine the an­swers to those ques­tions, but there are much more rigor­ous for­mu­la­tions in the same se­man­tic neigh­bor­hood that we can put to very good use.

So if we mostly just mean ‘ab­stract’ and ‘con­crete,’ why talk about ‘log­i­cal’ and ‘phys­i­cal’ at all? Well, I think EY is try­ing to con­strain what sorts of ab­stract and con­crete posits we take se­ri­ously. Var­i­ous con­cepts of God, for in­stance, qual­ify as ‘ab­stract’ in the sense that they are not spa­tial; and psy­chic rays qual­ify as ‘con­crete’ in the sense that they oc­cur in spe­cific places; but based on a va­ri­ety of prin­ci­ples (e.g., ‘ab­stract things have no causal power in their own right’ and ‘con­crete things do not travel faster than light’ or ‘con­crete things are not ir­re­ducibly “men­tal”’), he seeks to tersely rule out the less re­al­is­tic spa­tial and non-spa­tial posits some peo­ple make, so that the epistemic grown-ups can have a more se­ri­ous dis­cus­sion amongst them­selves.

• he adopts the con­ven­tion that suffi­ciently care­ful purely ab­stract rea­son­ing (i.e., rea­son­ing with­out rea­son­ing about any par­tic­u­lar spa­tiotem­po­ral thing or pat­tern) is ‘log­i­cal,’

If this is the case, then I think he has failed to show that moral­ity is logic, un­less he’s us­ing an ex­tremely lax stan­dard of “suffi­ciently care­ful”. For ex­am­ple, I think that “suffi­ciently care­ful” rea­son­ing must at a min­i­mum be us­ing a method of rea­son­ing that is not sen­si­tive to the or­der in which one en­coun­ters ar­gu­ments, and is not sen­si­tive to the mood one is in when con­sid­er­ing those ar­gu­ments. Do you think Eliezer has shown this? Or al­ter­na­tively, what stan­dard of “suffi­ciently care­ful” do you think Eliezer is us­ing when he says “moral­ity is logic”?

• I’d split up Eliezer’s view into sev­eral dis­tinct claims:

1. A se­man­tic the­sis: Log­i­cally reg­i­mented ver­sions of fair­ness, harm, obli­ga­tion, etc. are rea­son­able se­man­tic can­di­dates for moral terms. They may not be what ev­ery­one ac­tu­ally means by ‘fair’ and ‘vir­tu­ous’ and so on, but they’re mod­est im­prove­ments in the same way that a rigor­ous genome-based defi­ni­tion of Ca­nis lu­pus fa­mil­iaris would be a rea­son­able im­prove­ment upon our ca­sual, ev­ery­day con­cept of ‘dog,’ or that a clear set of ther­mo­dy­namic thresh­olds would be a rea­son­able reg­i­men­ta­tion of our ev­ery­day con­cept ‘hot.’

2. A meta­phys­i­cal the­sis: Th­ese reg­i­men­ta­tions of moral terms do not com­mit us to im­plau­si­ble mag­i­cal ob­jects like Div­ine Com­mands or Irre­ducible ‘Ought­ness’ Prop­er­ties In Our Fun­da­men­tal Physics. All they com­mit us to are the or­di­nary ob­jects of physics, logic, and math­e­mat­ics, e.g., sets, func­tions, and causal re­la­tion­ships; and sets, func­tions, and causal­ity are not meta­phys­i­cally ob­jec­tion­able.

3. A nor­ma­tive the­sis: It is use­ful to adopt moral­i­tys­peak our­selves, pro­vided we do so us­ing a use­fully reg­i­mented se­man­tics. The rea­sons to re­fuse to talk in a moral idiom are, in part thanks to 1 and 2, not strong enough to out­weigh the rhetor­i­cal and self-mo­ti­va­tional ad­van­tages of adopt­ing such an idiom.

It seems clear to me that you dis­agree with the­sis 1; but if you granted 1 (e.g., granted that ‘a func­tion that takes in­equitable dis­tri­bu­tions of re­sources be­tween equally de­serv­ing agents into equitable dis­tri­bu­tions thereof’ is not a crazy can­di­date mean­ing for the English word ‘fair­ness’), would you still dis­agree with 2 and 3? And do you think that moral­ity is un­usual in failing 1-style reg­i­men­ta­tion, or do you think that we’ll even­tu­ally need to ditch nearly all English-lan­guage terms if we are to at­tain rigor?

• I like this splitup!

(From the great-grand­par­ent.)

Eliezer’s stan­dard use of ‘log­i­cal’ takes the ‘ab­stract’ part of log­i­cal­ish vibes and runs with them; he adopts the con­ven­tion that suffi­ciently care­ful purely ab­stract rea­son­ing (i.e., rea­son­ing with­out rea­son­ing about any par­tic­u­lar spa­tiotem­po­ral thing or pat­tern) is ‘log­i­cal,’ whereas rea­son­ing about con­crete things-in-the-world is ‘phys­i­cal.’

I think I want to make a slightly stronger claim than this; i.e. that by log­i­cal dis­course we’re thin­ning down a uni­verse of pos­si­ble mod­els us­ing ax­ioms.

One thing I didn’t go into, in this episte­mol­ogy se­quence, is the no­tion of ‘effec­tive­ness’ or ‘for­mal­ity’, which is im­por­tant but I didn’t go into as much be­cause my take on it feels much more stan­dard—I’m not sure I have any­thing more to say about what con­sti­tutes an ‘effec­tive’ for­mula or ax­iom or com­pu­ta­tion or phys­i­cal de­scrip­tion than other work­ers in the field. This car­ries a lot of the load in prac­tice in re­duc­tion­ism; e.g., the prob­lem with ir­re­ducible fear is that you have to ap­peal to your own brain’s na­tive fear mechanisms to carry out pre­dic­tions about it, and you can never write down what it looks like. But af­ter we’re done be­ing effec­tive, there’s still the ques­tion of whether we’re nav­i­gat­ing to a part of the phys­i­cal uni­verse, or nar­row­ing down math­e­mat­i­cal mod­els, and by ‘log­i­cal’ I mean to re­fer to the lat­ter sort of thing rather than the former. The load of talk­ing about suffi­ciently care­ful rea­son­ing is mostly car­ried by ‘effec­tive’ as dis­t­in­guished from em­pa­thy-based pre­dic­tions, ap­peals to im­plicit knowl­edge, and so on.

I also don’t claim to have given moral­ity an effec­tive de­scrip­tion—my ac­tual moral ar­gu­ments gen­er­ally con­sist in ap­peal­ing to im­plicit and hope­fully shared rea­sons-for-ac­tion, not deriva­tions from ax­ioms—but the meta­phys­i­cal and nor­ma­tive claim is that these rea­sons-for-ac­tion both have an effec­tive de­scrip­tion (de­scrip­tively speak­ing) and that any ideal­ized or nor­ma­tive ver­sion of them would still have an effec­tive de­scrip­tion (nor­ma­tively speak­ing).

• Let me try a differ­ent tack in my ques­tion­ing, as I sus­pect maybe your claim is along a differ­ent axis than the one I de­scribed in the sibling com­ment. So far you’ve in­tro­duced a bunch of “mov­ing parts” for your metaeth­i­cal the­ory:

• moral arguments

• im­plicit rea­sons-for-action

• effec­tive de­scrip­tions of rea­sons-for-action

• util­ity function

But I don’t un­der­stand how these are sup­posed to fit to­gether, in an al­gorith­mic sense. In de­ci­sion the­ory we also have miss­ing mod­ules or black boxes, but at least we spec­ify their types and how they in­ter­act with the other com­po­nents, so we can have some con­fi­dence that ev­ery­thing might work once we fill in the blanks. Here, what are the types of each of your pro­posed metaeth­i­cal ob­jects? What’s the “con­trol­ling al­gorithm” that takes moral ar­gu­ments and im­plicit rea­sons-for-ac­tion, and pro­duces effec­tive de­scrip­tions of rea­sons-for-ac­tion, and even­tu­ally the fi­nal util­ity func­tion?

As you ar­gued in Un­nat­u­ral Cat­e­gories (which I keep cit­ing re­cently), rea­sons-for-ac­tion can’t be re­duced the same way as nat­u­ral cat­e­gories. But it seems com­pletely opaque to me how they are sup­posed to be re­duced, be­sides that moral ar­gu­ments are in­volved.

Am I ask­ing for too much? Per­haps you are just say­ing that these must be the rele­vant parts, and let’s figure out both how they are sup­posed to work in­ter­nally, and how they are sup­posed to fit to­gether?

• my ac­tual moral ar­gu­ments gen­er­ally con­sist in ap­peal­ing to im­plicit and hope­fully shared rea­sons-for-ac­tion, not deriva­tions from axioms

So would it be fair to say that your ac­tual moral ar­gu­ments do not con­sist of suffi­ciently care­ful rea­son­ing?

these rea­sons-for-ac­tion both have an effec­tive de­scrip­tion (de­scrip­tively speak­ing)

Is there a differ­ence be­tween this claim and the claim that our ac­tual cog­ni­tion about moral­ity can be de­scribed as an al­gorithm? Or are you say­ing that these rea­sons-for-ac­tion con­sti­tute (cur­rently un­known) ax­ioms which to­gether form a con­sis­tent log­i­cal sys­tem?

Can you see why I might be con­fused? The former in­ter­pre­ta­tion is too weak to dis­t­in­guish moral­ity from any­thing else, while the lat­ter seems too strong given our cur­rent state of knowl­edge. But what else might you be say­ing?

any ideal­ized or nor­ma­tive ver­sion of them would still have an effec­tive de­scrip­tion (nor­ma­tively speak­ing).

Similar ques­tion here. Any you say­ing any­thing be­yond that any ideal­ized or nor­ma­tive way of think­ing about moral­ity is still an al­gorithm?

• but if you granted 1 (e.g., granted that ‘a func­tion that takes in­equitable dis­tri­bu­tions of re­sources be­tween equally de­serv­ing agents into equitable dis­tri­bu­tions thereof’ is not a crazy can­di­date mean­ing for the English word ‘fair­ness’), would you still dis­agree with 2 and 3?

If I grant 1, I cur­rently can’t think of any ob­jec­tions to 2 and 3 (which doesn’t mean that I won’t if I took 1 more se­ri­ously and there­fore had more in­cen­tive to look for such ob­jec­tions).

And do you think that moral­ity is un­usual in failing 1-style reg­i­men­ta­tion, or do you think that we’ll even­tu­ally need to ditch nearly all English-lan­guage terms if we are to at­tain rigor?

I think at a min­i­mum, it’s un­usu­ally difficult to do 1-style reg­i­men­ta­tion for moral­ity (and Eliezer him­self ex­plained why in Un­nat­u­ral Cat­e­gories). I guess one point I’m try­ing to make is that what­ever kind of rea­son­ing we’re us­ing to at­tempt this kind of reg­i­men­ta­tion is not the same kind of rea­son­ing that we use to think about some log­i­cal ob­ject af­ter we have reg­i­mented it. Does that make sense?

• Rob­bBB prob­a­bly knows this, but I’d just like to men­tion that the three claims listed above, at least as stated there, are com­mon to many metaeth­i­cal ap­proaches, not just Eliezer’s. De­sirism is one ex­am­ple. Other ex­am­ples in­clude the moral re­duc­tion­isms of Richard Brandt, Peter Rail­ton, and Frank Jack­son.

• By “moral­ity” you seem to mean some­thing like ‘the set of judg­ments about mass wellbe­ing or­di­nary un­trained hu­mans ar­rive at when prompted.’ This is about like deny­ing the pos­si­bil­ity of ar­ith­metic be­cause peo­ple sys­tem­at­i­cally make er­rors in math­e­mat­i­cal rea­son­ing. When the Pythagore­ans rea­soned about num­bers, they were not be­ing ‘suffi­ciently care­ful;’ they did not rigor­ously define what it took for some­thing to be a num­ber or to have a solu­tion, or stipu­late ex­actly what op­er­a­tions are pos­si­ble; and they did not have a clear no­tion of the ab­stract/​con­crete dis­tinc­tion, or of which of these two do­mains ‘num­ber’ should be­long to. Quite plau­si­bly, Pythagore­ans would ar­rive at differ­ent solu­tions in some cases based on their state of mind or the prob­lems’ fram­ing; and cer­tainly Pythagore­ans ran into dis­agree­ments they could not re­solve and fell into war­ring camps as a re­sult, e.g., over whether there are ir­ra­tional num­bers.

But the un­rea­son­able­ness of the dis­putants, no mat­ter how ex­treme, can­not in­fect the sub­ject mat­ter and make that sub­ject mat­ter in­trin­si­cally im­pos­si­ble to care­fully rea­son with. No mat­ter how ex­treme we make the Pythagore­ans’ ec­cen­tric­i­ties, as long as they con­tinue to do some­thing math-ish, it would re­main pos­si­ble for a Eu­clid or Yud­kowsky to arise from the sea-foam and pro­pose a reg­i­men­ta­tion of their in­tu­itions, a more care­fully for­mal­ized ver­sion of their con­cepts of ‘num­ber,’ ‘ra­tio,’ ‘proof,’ etc.

I take it that Eliezer thinks we are very much in the po­si­tion to­day of in­hab­it­ing a global, heav­ily schis­ma­tized net­work of Pythagorean Cults of Mo­ral­ity. Those cults are ir­ra­tional, and their fa­vored con­cepts would need to be made more pre­cise and care­ful be­fore the ques­tions they ask could be as­signed de­ter­mi­nate an­swers (even in prin­ci­ple). But the sub­ject mat­ter those cults are talk­ing about—how to cul­ti­vate hu­man well-be­ing, how to dis­tribute re­sources equitably, how to bal­ance prefer­ences in a way most peo­ple would pre­fer, etc. -- is not in­trin­si­cally ir­ra­tional or mys­ti­cal or in­ef­fable. The cat­e­gories in ques­tion are track­ing real prop­erty clusters, though per­haps not yet with com­plete ap­pli­ca­bil­ity-to-any-old-case; no mat­ter how much of a moral anti-re­al­ist you are, for in­stance, you can’t rea­son­ably hold that ‘fair­ness’ doesn’t have its own set of satis­fac­tion con­di­tions that fail to co­in­cide with other moral (or phys­i­cal, math­e­mat­i­cal, etc.) con­cepts.

Another way of mo­ti­vat­ing the idea that moral­ity is ‘log­i­cal’: De­ci­sion the­ory is ‘log­i­cal’, and moral­ity is a spe­cial sort of de­ci­sion the­ory. If we can care­fully reg­i­ment the satis­fac­tion con­di­tions for an in­di­vi­d­ual’s prefer­ences, then we can reg­i­ment the satis­fac­tion con­di­tions for the prefer­ences of peo­ple gen­er­ally; and we can iso­late the prefer­ences that peo­ple con­sider moral vs. amoral; and if we can do all that, what skep­ti­cal challenge could block an al­gorithm that rec­og­niz­ably maps what we call ‘fair’ and ‘un­fair’ and ‘moral’ and ‘im­moral,’ that couldn’t equally well block an al­gorithm that rec­og­niz­ably maps what we call ‘preferred’ and ‘dis­taste­ful’ and ‘deli­cious’...? How care­lessly do peo­ple have to rea­son with x such that we can con­clude that it’s im­pos­si­ble to rea­son care­fully with x?

• But the un­rea­son­able­ness of the dis­putants, no mat­ter how ex­treme, can­not in­fect the sub­ject mat­ter and make that sub­ject mat­ter in­trin­si­cally im­pos­si­ble to care­fully rea­son with.

I think I’ve been care­ful not to claim that moral­ity is im­pos­si­ble to care­fully rea­son with, but just that we don’t know how to care­fully rea­son with it yet and given our cur­rent state of knowl­edge, it may turn out to be im­pos­si­ble to care­fully rea­son with.

Another way of mo­ti­vat­ing the idea that moral­ity is ‘log­i­cal’: De­ci­sion the­ory is ‘log­i­cal’, and moral­ity is a spe­cial sort of de­ci­sion the­ory.

With de­ci­sion the­ory, we’re also in a “non-log­i­cal” state of rea­son­ing, where we don’t yet have a log­i­cal defi­ni­tion of what con­sti­tutes cor­rect de­ci­sion the­ory and there­fore can’t just ap­ply log­i­cal rea­son­ing. What’s helpful in the case of de­ci­sion the­ory is that it seems rea­son­able to as­sume that when we do come up with such a log­i­cal defi­ni­tion, it will be rel­a­tively sim­ple. This helps tremen­dously in guid­ing our search, and partly com­pen­sates for the fact that we do not know how to rea­son care­fully dur­ing this search. But with “moral­ity”, we don’t have this crutch since we think it may well be the case that “value is com­plex”.

• we don’t know how to care­fully rea­son with it yet and given our cur­rent state of knowl­edge, it may turn out to be im­pos­si­ble to care­fully rea­son with.

I agree that it’s go­ing to take a lot of work to fully clar­ify our con­cepts. I might be able to as­sign a less re­mote prob­a­bil­ity to ‘moral­ity turns out to be im­pos­si­ble to care­fully rea­son with’ if you could give an ex­am­ple of a similarly com­plex hu­man dis­course that turned out in the past to be ‘im­pos­si­ble to care­fully rea­son with’.

High-qual­ity the­ol­ogy is an ex­am­ple of the op­po­site; we turned out to be able to rea­son very care­fully (though ad­mit­tedly most the­ol­ogy is sub­par) with slightly reg­i­mented ver­sions of con­cepts in nat­u­ral re­li­gion. At least, there are some cases where the reg­i­men­ta­tion was not com­pletely per­verse, though the cra­zier ex­am­ples may be more salient in our mem­o­ries. But the biggest prob­lem with was meta­phys­i­cal, not se­man­tic; there just weren’t any things in the neigh­bor­hood of our cat­e­gories for us to re­fer to. If you have no meta­phys­i­cal ob­jec­tions to Eliezer’s treat­ment of moral­ity be­yond your se­man­tic ob­jec­tions, then you don’t think a reg­i­mented moral­ity would be prob­le­matic for the rea­sons a reg­i­mented the­ol­ogy would be. So what’s a bet­ter ex­am­ple of a reg­i­men­ta­tion that would fail be­cause we just can’t be care­ful about the topic in ques­tion? What symp­toms and causes would be di­ag­nos­tic of such cases?

What’s helpful in the case of de­ci­sion the­ory is that it seems rea­son­able to as­sume that when we do come up with such a log­i­cal defi­ni­tion, it will be rel­a­tively sim­ple.

By com­par­i­son, per­haps. But it de­pends a whole lot on what we mean by ‘moral­ity’. For in­stance, do we mean:?

• Mo­ral­ity is the hy­po­thet­i­cal de­ci­sion pro­ce­dure that, if fol­lowed, tends to max­i­mize the amount of pos­i­tively valenced ex­pe­rience in the uni­verse rel­a­tive to nega­tively valenced ex­pe­rience, to a greater ex­tent than any other de­ci­sion pro­ce­dure.

• Mo­ral­ity is the hy­po­thet­i­cal de­ci­sion pro­ce­dure that, if fol­lowed, tends to max­i­mize the oc­cur­rence of states of af­fairs that agents pre­fer rel­a­tive to states they do not pre­fer (tak­ing into ac­count that agents gen­er­ally pre­fer not to have their prefer­ences rad­i­cally al­tered).

• Mo­ral­ity is any de­ci­sion pro­ce­dure that any­one wants peo­ple in gen­eral to fol­low.

• Mo­ral­ity is the hu­man ten­dency to con­struct and pre­scribe rules they want peo­ple in gen­eral to fol­low.

• Mo­ral­ity is any­thing that English-lan­guage speak­ers call “moral­ity” with a cer­tain high fre­quency.

If “value is com­plex,” that’s a prob­lem for pru­den­tial de­ci­sion the­o­ries based on in­di­vi­d­ual prefer­ences, just as much as it is for agent-gen­eral moral de­ci­sion the­o­ries. But I think we agree both there’s a long way to go in reg­i­ment­ing de­ci­sion the­ory, and that there’s some ini­tial plau­si­bil­ity and util­ity in try­ing to reg­i­ment a mor­al­iz­ing class of de­ci­sion the­o­ries; whether we call this reg­i­ment­ing pro­ce­dure ‘logi­ciz­ing’ is just a ter­minolog­i­cal is­sue.

• But it de­pends a whole lot on what we mean by ‘moral­ity’.

What I mean by “moral­ity” is the part of nor­ma­tivity (“what you re­ally ought, all things con­sid­ered, to do”) that has to do with val­ues (as op­posed to ra­tio­nal­ity).

I agree that it’s go­ing to take a lot of work to fully clar­ify our con­cepts. I might be able to as­sign a less re­mote prob­a­bil­ity to ‘moral­ity turns out to be im­pos­si­ble to care­fully rea­son with’ if you could give an ex­am­ple of a similarly com­plex hu­man dis­course that turned out in the past to be ‘im­pos­si­ble to care­fully rea­son with’.

In gen­eral, I’m not sure how to show a nega­tive like “it’s im­pos­si­ble to rea­son care­fully about sub­ject X”, so the best I can do is ex­hibit some sub­ject that peo­ple don’t know how to rea­son care­fully about and in­tu­itively seems like it may be im­pos­si­ble to rea­son care­fully about. Take the ques­tion, “Which sets re­ally ex­ist?” (Do large car­di­nals ex­ist, for ex­am­ple?) Is this a con­vinc­ing ex­am­ple to you of an­other sub­ject that may be im­pos­si­ble to rea­son care­fully about?

• I take it that Eliezer thinks we are very much in the po­si­tion to­day of in­hab­it­ing a global, heav­ily schis­ma­tized net­work of Pythagorean Cults of Mo­ral­ity. Those cults are ir­ra­tional, and their fa­vored con­cepts would need to be made more pre­cise and care­ful be­fore the ques­tions they ask could be as­signed de­ter­mi­nate an­swers (even in prin­ci­ple). But the sub­ject mat­ter those cults are talk­ing about—how to cul­ti­vate hu­man well-be­ing, how to dis­tribute re­sources equitably, -- is not in­trin­si­cally ir­ra­tional or mys­ti­cal or in­ef­fable. The cat­e­gories in ques­tion are track­ing real prop­erty clusters, though per­haps not yet with com­plete ap­pli­ca­bil­ity-to-any-old-case; no mat­ter how much of a moral anti-re­al­ist you are, for in­stance, you can’t rea­son­ably hold that ‘fair­ness’ doesn’t have its own set of satis­fac­tion con­di­tions that fail to co­in­cide with other moral (or phys­i­cal, math­e­mat­i­cal, etc.) con­cepts.

Haven’t we been in this po­si­tion since be­fore math­e­mat­ics was a thing. The lack of progress to­wards con­sen­sus in that pe­riod of time seems dis­heart­en­ing.

• The nat­u­ral num­ber line is one of the sim­plest struc­tures a hu­man be­ing is ca­pa­ble of con­ceiv­ing. The idea of a hu­man prefer­ence is one of the most com­plex struc­tures a hu­man be­ing has yet en­coun­tered. And we have a lot more emo­tional in­vest­ment and evolu­tion­ary bag­gage in­terfer­ing with care­fully ax­io­m­a­tiz­ing our prefer­ences than with care­fully ax­io­m­a­tiz­ing the num­bers. Why should we be sur­prised that we’ve made more progress with reg­i­ment­ing num­ber the­ory than with reg­i­ment­ing moral­ity or de­ci­sion the­ory in the last few thou­sand years?

• In terms of moral the­ory, we ap­pear to have made no progress at all. We don’t even agree on defi­ni­tions.

Math­e­mat­ics may or might not be an em­piri­cal dis­ci­pline, but if you get your math wrong badly enough, you lose the abil­ity to pay rent.

If moral­ity paid rent in an­ti­ci­pated ex­pe­rience, I’d ex­pect so­cieties that had more cor­rect moral­ity to do bet­ter and so­cieties with less cor­rect moral­ity to do worse. Mo­ral­ity is so im­por­tant that I ex­pect marginal differ­ences to have ma­jor im­pact. And I just don’t see the ev­i­dence that such an im­pact is or ever did hap­pen.

So, have I mis­read his­tory? Or have I made a mis­take in pre­dict­ing that chance differ­ences in moral­ity should have ma­jor im­pacts on the pros­per­ity of a so­ciety? (Or some other er­ror?)

• In terms of moral the­ory, we ap­pear to have made no progress at all. We don’t even agree on defi­ni­tions.

But defin­ing terms is the triv­ial part of any the­ory; if you con­cede that we haven’t even got­ten that far (and that term-defin­ing is triv­ial), then you’ll have a much harder time ar­gu­ing that if we did agree on defi­ni­tions we’d still have made no progress. You can’t ar­gue that, be­cause if we all have differ­ing term defi­ni­tions, then that on its own pre­dicts rad­i­cal dis­agree­ment about al­most any­thing; there is no need to posit a fur­ther ex­pla­na­tion.

If moral­ity paid rent in an­ti­ci­pated experience

Mo­ral­ity pays rent in an­ti­ci­pated ex­pe­rience in the same three ba­sic ways that math­e­mat­ics does:

1. Know­ing about moral­ity helps us pre­dict the be­hav­ior of moral­ists, just as know­ing about math­e­mat­ics helps us pre­dict the be­hav­ior of math­e­mat­i­ci­ans (in­clud­ing their cre­ations). If you know that peo­ple think mur­der is bad, you can help pre­dict why mur­der is so rare; just as know­ing math­e­mat­i­ci­ans’ be­liefs about nat­u­ral num­bers helps us pre­dict what funny squig­gly lines will oc­cur on calcu­la­tors. This, of course, doesn’t re­quire any com­mit­ment to moral re­al­ism, just as it doesn’t re­quire a com­mit­ment to math­e­mat­i­cal re­al­ism.

2. Inas­much as the struc­ture of moral rea­son­ing mir­rors the struc­ture of phys­i­cal sys­tems, we can pre­dict how phys­i­cal sys­tems will change based on what our moral ax­ioms out­put. For in­stance, if our moral ax­ioms are care­fully tuned to par­allel the dis­tri­bu­tion of suffer­ing in the world, we can use them to pre­dict what sorts of brain-states will be phys­i­cally in­stan­ti­ated if we perform cer­tain be­hav­iors. Similarly, if our num­ber ax­ioms are care­fully tuned to par­allel the changes in phys­i­cal ob­jects (and heaps thereof) in the world, we can use them to pre­dict how phys­i­cal ob­jects will change when we trans­late them in space­time.

3. Inas­much as our in­tu­itions give rise to our con­vic­tions about math­e­mat­ics and moral­ity, we can use the afore­men­tioned con­vic­tions to pre­dict our own fu­ture in­tu­itions. In par­tic­u­lar, an es­pe­cially reg­i­mented math­e­mat­ics or moral­ity, that arises from highly in­tu­itive ax­ioms we ac­cept, will of­ten al­low us to al­gorith­mi­cally gen­er­ate what we would re­flec­tively find most in­tu­itive be­fore we can even pro­cess the in­for­ma­tion suffi­ciently to gen­er­ate the in­tu­ition. A calcu­la­tor gives us the most in­tu­itive and re­flec­tively sta­ble value for 142857 times 7 be­fore we’ve gone to the trou­ble of un­der­stand­ing why or that this is the most in­tu­itive value; similarly, a suffi­ciently ad­vanced util­ity-calcu­la­tor, pro­grammed with the rules you find most re­flec­tively in­tu­itive, would gen­er­ate the ul­ti­mately in­tu­itive an­swers for moral dilem­mas be­fore you’d even gone to the trou­ble of figur­ing out on your own what you find most in­tu­itive. And your fu­ture in­tu­itions are fu­ture ex­pe­riences; so the propo­si­tions of math­e­mat­ics and moral­ity, in­ter­est­ingly enough, serve as pre­dic­tors for your own fu­ture men­tal states, at least when those men­tal states are suffi­ciently care­ful and thought out.

But all of these are to some ex­tent in­di­rect. It’s not as though we di­rectly ob­serve that SSSSSSS0 is prime, any more than we di­rectly ob­serve that mur­der is bad. We ei­ther take it as a given, or de­rive it from some­thing else we take as a given; but re­gard­less, there can be plenty of in­di­rect ways that the ‘log­i­cal’ dis­course in ques­tion helps us bet­ter nav­i­gate, ma­nipu­late, and pre­dict our en­vi­ron­ments.

If moral­ity paid rent in an­ti­ci­pated ex­pe­rience, I’d ex­pect so­cieties that had more cor­rect moral­ity to do bet­ter and so­cieties with less cor­rect moral­ity to do worse.

There’s a prob­lem here: What are we us­ing to eval­u­ate ‘do­ing bet­ter’ vs. ‘do­ing worse’? We of­ten use moral su­pe­ri­or­ity it­self as an im­por­tant mea­sure of ‘bet­ter­ness;’ we think it’s morally right or op­ti­mal to max­i­mize hu­man well-be­ing, so we judge so­cieties that do a good job of this as ‘bet­ter.’ At the very least, moral con­sid­er­a­tions like this seem to be a part of what we mean by ‘bet­ter.’ If you’re try­ing to bracket that kind of suc­cess, then it’s not clear to me what you even mean by ‘bet­ter’ or ‘pros­per­ity’ here. Are you ask­ing whether moral for­ti­tude cor­re­lates with GDP?

• (Some com­mon senses of “moral for­ti­tude” definitely cause GDP, at min­i­mum in the form of trust be­tween busi­ness­peo­ple and less preda­tory bu­reau­crats. But this part is equally true of Babyeaters.)

• There’s a pseudo-the­o­rem in math that is some­times given to 1st year grad­u­ate stu­dents (at least in my case, 35 years ago), which is that

All nat­u­ral num­bers are in­ter­est­ing.

Nat­u­ral num­bers con­sist of {1, 2, 3, …} -- ac­tu­ally a re­cent hot topic of con­ver­sa­tion on LW (“nat­u­ral num­bers” is some­times defined to in­clude 0, but ev­ery­thing that fol­lows will work ei­ther way).

The “proof” used the prin­ci­ple of math­e­mat­i­cal in­duc­tion (one ver­sion of which is):

If P(n) is true for n=1, and the as­ser­tion “m is the small­est in­te­ger such that !P(m)” leads to a con­tra­dic­tion, then P(n) is true for all nat­u­ral num­bers.

and also uses the fact (from the Peano con­struc­tion of the nat­u­ral num­bers?) that ev­ery non-empty sub­set of nat­u­ral num­bers has a small­est el­e­ment.

PROOF:

1 is in­ter­est­ing.

Sup­pose the­o­rem is false. Then some num­ber m is the small­est un­in­ter­est­ing num­ber. But then wouldn’t that be in­ter­est­ing?

Con­tra­dic­tion. QED.

The illus­trates a pit­fall of mix­ing (qual­ities that don’t re­ally be­long in a math­e­mat­i­cal state­ment) with (rigor­ous logic), and in gen­eral, if you take a qual­ity that is not rigor­ously defined, and ap­ply a suffi­ciently long train of logic to it, you are li­able to “prove” non­sense.

(Note: the logic just ap­plied is equiv­a­lent to P(1) ⇒ P(2) ⇒ P(3), …which is in­finite and hence long enough.)

It is my im­pres­sion that cer­tain con­tested (though “proven”) as­ser­tions about eco­nomics suffer from this prob­lem, and it’s hard, for me at least, to think of a moral propo­si­tion that wouldn’t risk this sort of pit­fall.

• Okay but if I hon­estly be­lieve that all nat­u­ral num­bers are in­ter­est­ing and thought of this proof as pretty val­idly match­ing my in­tu­itions, what does that mean?

• Un­less you turn “in­ter­est­ing” into some­thing rigor­ously defined and pre­cisely com­mu­ni­cated to oth­ers, what it means is that all nat­u­ral num­bers are {some qual­ity that is not rigor­ously defined and can’t be pre­cisely com­mu­ni­cated to oth­ers}.

• I guess I feel that even if I haven’t defined “in­ter­est­ing” rigor­ously, I still have some in­tu­itions for what “in­ter­est­ing” means, large parts of which will be shared by my in­tended au­di­ence.

For ex­am­ple, I could make the em­piri­cal pre­dic­tion that if some­one names a num­ber I could talk about it for a bit and then they would agree it was in­ter­est­ing (I mean this as a toy ex­am­ple; I’m not sure I could do this.)

One could then take ap­prox­i­ma­tions of these con­ver­sa­tions, or even the ex­is­tence of these con­ver­sa­tions, and define in­ter­est­ing* to be “I can say a unique few sen­tences about his­toric re­sults sur­round­ing this num­ber and re­lated math­e­mat­i­cal fac­toids.” Which then might be a strong em­piri­cal pre­dic­tor of peo­ple claiming some­thing is in­ter­est­ing.

So I feel like there’s some­thing be­yond a use­less log­i­cal fact be­ing ex­pressed by my in­tu­itions here.

• I can’t tell what this is. The first link might im­ply that Gw­ern thinks I mis­stated the In­ter­est­ing Num­ber Para­dox (I looked at the Wikipe­dia ar­ti­cle be­fore I wrote my post, but went with my mem­ory, and there are mul­ti­ple equiv­a­lent ways of say­ing it, but if you think I got it wrong ….? Or maybe it was offered as a handy refer­ence.

The Berry Para­dox sounds like a very differ­ent ket­tle of fish … with more real com­plex­ity.

• Or maybe it was offered as a handy refer­ence.

I would bet on this one.

More meta: Per­haps your pri­ors for “if some­one replies to my com­ment, they dis­agree with me” are too high. ;-) Maybe not for in­ter­net in gen­eral, but LW is not an av­er­age in­ter­net site.

• In a re­cent post, Eliezer said “moral­ity is logic”

The ac­tual quote is:

moral­ity is (and should be) logic, not physics

• Peo­ple do all sorts of sloppy rea­son­ing; ev­ery­day logic also ar­rives at both A and ~A ; any sort of fuzzi­ness leads to that. To ac­tu­ally be moral, it is nec­es­sary that you can’t ar­rive at both A and ~A at will—oth­er­wise your moral­ity pro­vides no con­straint.

• Differ­ent peo­ple can dis­agree about pretty much any moral ques­tion. Any one per­son’s moral­ity may be sta­ble enough not to ar­rive at A and also ~A, but since the re­sult still de­pen­dent most of all on that per­son’s up­bring­ing and cul­turally en­dorsed be­lief, moral­ity is not very use­ful as logic. (Of course it is use­ful as moral­ity: our brains are built that way.)

• Differ­ence in val­ues is a lit­tle over­stated, I think. Prac­ti­cally, there’s lit­tle differ­ence be­tween what peo­ple say they’d do in Mil­gram ex­per­i­ment, but a huge differ­ence be­tween what they ac­tu­ally do.

• I’m not sure how to parse your gram­mar.

Are you say­ing that differ­ent peo­ple all say they will do the same (‘good’) thing on Mil­gram, but in prac­tice differ­ent peo­ple do differ­ent things on Mil­gram (some ‘good’ some ‘bad’)?

Or are you say­ing that there is a large differ­ence be­tween what peo­ple say they would do on Mil­gram, and be­tween what they ac­tu­ally do?

(Be­cause repli­ca­tions of Mil­gram are pro­hibited by mod­ern ethics boards, the data is weaker than I’d like it to be.)

You also say that I over­state the differ­ence in val­ues be­tween peo­ple. But Mil­gram ran his ex­per­i­ment just once on very ho­moge­nous peo­ple: all from the same cul­ture. If he’d com­pared it to widely differ­ing cul­tures, I ex­pect at least some of the time the com­pli­ance rates would differ sig­nifi­cantly.

• For all we know, some­body try­ing to rea­son about a moral con­cept like “fair­ness” may just be tak­ing a ran­dom walk as they move from one con­clu­sion to an­other based on moral ar­gu­ments they en­counter or think up.

Well. Not a purely ran­dom walk. A weighted one.

Isn’t this true of all be­liefs? And isn’t ra­tio­nal­ity just in­creas­ing the weight in the right di­rec­tion?

• The word “moral­ity” needs to be made more spe­cific for this dis­cus­sion. One of the things you seem to be talk­ing about is men­tal be­hav­ior that pro­duces value judg­ments or their jus­tifi­ca­tions. It’s some­thing hu­man brains do, and we can in prin­ci­ple sys­tem­at­i­cally study this hu­man ac­tivity in de­tail, or ab­stractly de­scribe hu­mans as brain ac­tivity al­gorithms and study those al­gorithms. This char­ac­ter­i­za­tion doesn’t seem par­tic­u­larly in­ter­est­ing, as you might also de­scribe math­e­mat­i­ci­ans in this way, but this won’t be any­where close to an effi­cient route to learn­ing about math­e­mat­ics or de­scribing what math­e­mat­ics is.

“Logic” and “math­e­mat­ics” are also some­what vague in this con­text. In one sense, “math­e­mat­ics” may re­fer to any­thing, as a way of con­sid­er­ing things, which makes the char­ac­ter­i­za­tion empty of con­tent. In an­other sense, it’s the study of the kinds of ob­jects that math­e­mat­i­ci­ans typ­i­cally study, but in this sense it prob­a­bly won’t re­fer to things like ac­tivity of hu­man brains or par­tic­u­lar phys­i­cal uni­verses. “Logic” is more spe­cific, it’s a par­tic­u­lar way of rep­re­sent­ing and pro­cess­ing math­e­mat­i­cal ideas. It al­lows de­scribing the things you are talk­ing about and ob­tain­ing new in­for­ma­tion about them that wasn’t ex­plicit in the origi­nal de­scrip­tion.

Mo­ral­ity in the FAI-rele­vant sense is a speci­fi­ca­tion of what to do with the world, and as such it isn’t con­cerned with hu­man cog­ni­tion. The ques­tion of the na­ture of moral­ity in this sense is a ques­tion about ways of spec­i­fy­ing what to do with the world. Such speci­fi­ca­tion would need to be able to do at least these two things: (1) it needs to be given with much less ex­plicit de­tail than what can be ex­tracted from it when de­ci­sions about novel situ­a­tions need to be made, which sug­gests that the study of logic might be rele­vant, and (2) it needs to be re­lated to the world, which sug­gests that the study of physics might be rele­vant.

This ques­tion about the na­ture of moral­ity is sep­a­rate from the ques­tion of how to pin­point the right speci­fi­ca­tion of moral­ity to use in a FAI, out of all pos­si­ble speci­fi­ca­tions. The difficulty of find­ing the right moral­ity seems mostly un­re­lated to de­scribing what kind of thing moral­ity is. If I put a note with a num­ber writ­ten on it in a box, it might be perfectly ac­cu­rate to say that the box con­tains a num­ber, even though it might be im­pos­si­ble to say what that num­ber is, pre­cisely, and even if peo­ple aren’t able to con­struct any in­ter­est­ing mod­els of the un­known num­ber.

• You are not us­ing the same defi­ni­tion of logic EY does. For him logic is ev­ery­thing that is not physics in his physics+logic (or ter­ri­tory+maps, in the pre­vi­ously pop­u­lar terms) pic­ture of the world. Math­e­mat­i­cal logic is a tiny sliver of what he calls “logic”. For com­par­i­son, in an in­stru­men­tal­ist de­scrip­tion there are ex­pe­riences+mod­els, and EY’s logic is roughly equiv­a­lent to “mod­els” (maps, in the map-ter­ri­tory du­al­ism), of which math­e­mat­ics is but one.

• 27 Dec 2012 12:52 UTC
2 points

Eliezer said “moral­ity is logic”, by which he seems to mean… well, I’m still not ex­actly sure what, but one in­ter­pre­ta­tion is that a per­son’s cog­ni­tion about moral­ity can be de­scribed as an al­gorithm, and that al­gorithm can be stud­ied us­ing log­i­cal rea­son­ing. (Which of course is true, but in that sense both math and liter­ary crit­i­cism as well as ev­ery other sub­ject of hu­man study would be logic.)

Thank you—I knew I ADBOCed with Eliezer’s meta-ethics, but I had trou­ble putting down in words the rea­son.

• 27 Dec 2012 0:08 UTC
2 points

With moral­ity though, we have no such method,

Every act of ly­ing is morally pro­hibited /​ This act would be a lie //​ This act is morally pro­hibited.

So here I have a bit of moral rea­son­ing, the con­clu­sion of which fol­lows from the premises. The ar­gu­ment is valid, so if the premises are true, the con­clu­sion can be con­sid­ered proven. So given that I can give you valid proofs for moral con­clu­sions, in what way is moral­ity not log­i­cal?

doesn’t have any of the nice prop­er­ties of that a well-con­structed sys­tem of logic would have, for ex­am­ple, con­sis­tency, val­idity, sound­ness...

The above ex­am­ple of moral rea­son­ing (as­sume for the sake of sim­plic­ity that this is my en­tire moral sys­tem) is con­sis­tant, and valid, and (if you ac­cept the premises) sound. Any­one who ac­cepts the premises must ac­cept the con­clu­sion. One might wa­ver on ac­cep­tance of the premises (this is true for ev­ery sub­ject) but the con­clu­sion fol­lows from them re­gard­less of what one’s mood is.

All that said, our moral rea­son­ing is of­ten fraught. But I don’t think makes moral­ity pe­cu­liar. The mis­takes we of­ten make with re­gard to moral rea­son­ing don’t seem to be differ­ent in kind from the mis­takes we make in, say, eco­nomics. Ethics, they say, is not an ex­act sci­ence.

• I should have given some ex­am­ples of the kind of moral rea­son­ing I’m refer­ring to.

• 1st link is am­bi­guity aver­sion.

Mo­ral­ity is com­monly taken to de­scribe what one will ac­tu­ally do when they are trad­ing off pri­vate gains vs other peo­ple’s losses. See this as ex­am­ple of moral judge­ment. Sup­pose Roberts is smarter. He will quickly see that he can donate 10% to char­ity, and it’ll take longer for him to rea­son about value of cash that was not given to him (rea­son­ing that may stop him from press­ing the but­ton), so there will be a tran­sient dur­ing which he pushes the but­ton, un­less he some­how sup­presses ac­tions dur­ing tran­sients. It’s an open ended prob­lem ‘un­like logic’ be­cause con­se­quences are difficult to eval­u­ate.

edit: been in a hurry.

• Ah, thank you, that is helpful.

In the case of ‘cir­cu­lar al­tru­ism’, I con­fess I’m quite at a loss. I’ve never re­ally man­aged to pull an ar­gu­ment out of there. But if we’re just talk­ing about the prac­tice of quan­tify­ing goods in moral judge­ments, then I agree with you there’s no strongly com­plete eth­i­cal calcu­lus that’s go­ing to do ren­der ethics a math­e­mat­i­cal sci­ence. But in at least in ‘cir­cu­lar rea­son­ing’ EY doesn’t need quite so strong a view: so far as I can tell, he’s just say­ing that our moral pas­sions con­flict with our re­flec­tive moral judge­ments. And even if we don’t have a strongly com­plete moral sys­tem, we can make log­i­cally co­her­ent re­flec­tive moral judge­ments. I’d go so far as to say we can make log­i­cally co­her­ent re­flec­tive liter­ary crit­i­cism judge­ments. Logic isn’t picky.

So while, on the one hand, I’m also (as yet) un­con­vinced about EY’s ethics, I think it goes too far in the op­po­site di­rec­tion to say that eth­i­cal rea­son­ing is in­her­ently fuzzy or illog­i­cal. Valid ar­gu­ments are valid ar­gu­ments, re­gard­less.

• Every act of ly­ing is morally pro­hibited /​ This act would be a lie //​ This act is morally pro­hibited.

That also ap­plies to liter­ary crit­i­cism: Wulky Wilk­insen shows colo­nial aliena­tion /​ Authors who show colo­nial alieni­a­tion are post-utopi­ans //​ Wulky Wilk­insen is a post-utopian.

• Every act of ly­ing is morally pro­hibited /​ This act would be a lie //​ This act is morally pro­hibited.

So here I have a bit of moral rea­son­ing, the con­clu­sion of which fol­lows from the premises.

The prob­lem is that when the con­clu­sion is “proven wrong” (i.e. “my gut tells me that it’s bet­ter to lie to an Al Qaeda prison guard than to tell him the launch codes for Amer­ica’s nu­clear weapons”), then the premises that you started with are wrong.

So if I’m un­der­stand­ing Wei_Lai’s point, it’s that the name of the game is to find a premise that can­not and will not be con­tra­dicted by other moral premises via a bizarre hy­po­thet­i­cal situ­a­tion.

I be­lieve that Sam Har­ris has already mas­tered this thought ex­per­i­ment. Para­phrased from his de­bate with William Lane Craig:

“There ex­ists a hy­po­thet­i­cal uni­verse in which there is the ab­solute most amount of suffer­ing pos­si­ble. Ac­tions that move us away from that uni­verse are con­sid­ered good; ac­tions that move us to­wards that uni­verse are con­sid­ered bad”.

• I be­lieve that Sam Har­ris has already mas­tered this thought ex­per­i­ment. Para­phrased from his de­bate with William Lane Craig:
”There ex­ists a hy­po­thet­i­cal uni­verse in which there is the ab­solute most amount of suffer­ing pos­si­ble. Ac­tions that move us away from that uni­verse are con­sid­ered good; ac­tions that move us to­wards that uni­verse are con­sid­ered bad”.

This is why I find Har­ris frus­trat­ing. He’s stat­ing some­thing pretty much ev­ery­one agrees with, but they all make differ­ent sub­sti­tu­tions for the vari­able “suffer­ing.” And then Har­ris is vague about what he per­son­ally plugs in.

• At least as para­phrased here, the defi­ni­tion of “move to­wards” is very un­clear. Is it a uni­verse with more suffer­ing? A uni­verse with more suffer­ing right now? A uni­verse with more net pre­sent suffer­ing, ac­cord­ing to some dis­count rate? What if I move to a uni­verse with more suffer­ing both right now and for all pos­si­ble fu­ture dis­count rates, as­sum­ing no fur­ther ac­tion, but for which fu­ture ac­tions that greatly re­duce suffer­ing are made eas­ier? (In other words, does this sys­tem get stuck in lo­cal op­ti­mums?)

I think there is much that this ap­proach fails to solve, even if we all agree on how to mea­sure suffer­ing.

(In­cluded in “how to mea­sure suffer­ing” is a bit of com­pli­cated stuff like av­er­age vs to­tal util­i­tar­i­anism, and how to han­dle ex­is­ten­tial risks, and how to do prob­a­bil­ity math on out­comes that pro­duce a like­li­hood of suffer­ing.)

• The prob­lem is that when the con­clu­sion is “proven wrong”...then the premises that you started with are wrong.

I hope so! It would be ter­ribly awk­ward to find our­selves with true premises, valid rea­son­ing, and a false con­clu­sion. But un­less by ‘gut feel­ing’ you mean a valid ar­gu­ment with true premises, then gut feel­ings can’t prove any­thing wrong.

So if I’m un­der­stand­ing Wei_Lai’s point, it’s that the name of the game is to find a premise that can­not and will not be con­tra­dicted by other moral premises via a bizarre hy­po­thet­i­cal situ­a­tion.

Per­haps, though that wouldn’t speak to whether or not moral­ity is log­i­cal. If Wai Dai’s point is that moral­ity is, at best, ax­io­matic, then sure. But so is Peano ar­ith­metic, and that’s as log­i­cal as can be.

• I just stum­bled into this dis­cus­sion af­ter read­ing an ar­ti­cle about why math­e­mat­i­ci­ans and sci­en­tists dis­like tra­di­tional, So­cratic philos­o­phy, and my mind­set is fresh off that ar­ti­cle.

It was a fan­tas­tic read, but the un­der­ly­ing theme that I feel is rele­vant to this dis­cus­sion is this:

• So­cratic philos­o­phy treats log­i­cal ax­ioms as “self-ev­i­dent truths” (i.e. I think, there­fore I am).

• Math­e­mat­ics treats log­i­cal ax­ioms as “propo­si­tions”, and uses logic to see where those propo­si­tions lead (i.e. if you have a line and a point, the num­ber/​amount of lines that you can draw through the point that’s par­allel to the origi­nal line de­ter­mines what type of ge­om­e­try you are work­ing with (mul­ti­di­men­sional, spher­i­cal, or flat-plane ge­om­e­try)).

• Scien­tists treat log­i­cal ax­ioms as “hy­pothe­ses”, and log­i­cal “con­clu­sions” as testable state­ments that can de­ter­mine whether those ax­ioms are true or not (i.e. if this weird sys­tem known as “quan­tum me­chan­ics” were true, then we would see an in­terfer­ence pat­tern when shoot­ing elec­trons through a screen with 2 slits).

So I guess the point that we should be mak­ing is this: which philo­soph­i­cal ap­proach to­wards logic should we take to study ethics? I be­lieve Wei_Lai would say that the first ap­proach, treat­ing eth­i­cal ax­ioms as “self-ev­i­dent truths” is prob­le­matic due to the fact that a lot of hy­po­thet­i­cal situ­a­tions (like my ex­am­ple be­fore) can cre­ate a lot of con­tra­dic­tions be­tween var­i­ous eth­i­cal ax­ioms (i.e. choos­ing be­tween tel­ling a lie and let­ting ter­ror­ists blow up the planet).

• In­ter­est­ing piece. I was a bit be­mused by this, though:

In fact Plato wrote to Archimedes, scold­ing him about mess­ing around with real lev­ers and ropes when any gen­tle­man would have stayed in his study or pos­si­bly, in Archimedes’ case, his bath.

Prob­le­mat­i­cally for the story, Plato died around 347 BCE, and Archimedes wasn’t born un­til 287 BCE—sixty years later.

• So­cratic philos­o­phy treats log­i­cal ax­ioms as “self-ev­i­dent truths” (i.e. I think, there­fore I am).

I read the ar­ti­cle. It’s in­ter­est­ing (I liked the thing about pegs and strings), but I don’t think the guy’s (nor you) read a lot of ac­tual Greek philos­o­phy. I don’t mean that as an at­tack (why would you want to, af­ter all?), but it makes some of his, and your claims a lit­tle strange.

Socrates, in the Pla­tonic di­alogues, is un­will­ing to take the law of non-con­tra­dic­tion as an ax­iom. There just aren’t any ax­ioms in So­cratic philos­o­phy, just dis­cus­sions. No proofs, just con­ver­sa­tions. Plato (and cer­tainly not Socrates) doesn’t have doc­trines, and Plato is to­tally and in­ten­tion­ally mer­ciless with peo­ple who try to find Pla­tonic doc­trines.

Also, Plato and Socrates pre­date, for most pur­poses, logic.

Math­e­mat­ics treats log­i­cal ax­ioms as “propo­si­tions”, and uses logic to see where those propo­si­tions lead

Right, Aris­to­tle largely in­vented (or dis­cov­ered) that trick. Aris­to­tle’s logic is con­sis­tant and strongly com­plete (i.e. it’s not ax­io­matic, and re­lies on no ex­ter­nal log­i­cal con­cepts). Eu­clid picked up on it, and pro­duced a com­plete and con­sis­tant math­e­mat­ics. So (some) Greek philos­o­phy cer­tainly shares this idea with mod­ern math­e­mat­ics.

Scien­tists treat log­i­cal ax­ioms as “hy­pothe­ses”, and log­i­cal “con­clu­sions” as testable state­ments that can de­ter­mine whether those ax­ioms are true or not

I don’t think sci­en­tists treat log­i­cal ax­ioms as hy­pothe­ses. Log­i­cal ax­ioms aren’t em­piri­cal claims, and aren’t re­ally sub­ject to test­ing. But Aris­to­tle’s work on biol­ogy, me­te­o­rol­ogy, etc. for­wards plenty of em­piri­cal hy­pothe­ses, along with em­piri­cal ev­i­dence for them. Tex­tual ev­i­dence sug­gests Aris­to­tle performed lots of ex­per­i­ments, mostly in the form of vivi­sec­tion of an­i­mals. He was wrong about pretty much ev­ery­thing, but his method was em­piri­cal.

This is to say noth­ing of con­tem­po­rary philos­o­phy, which cer­tainly doesn’t take very much as ‘self-ev­i­dent truth’. I can as­sure you, no one gets any­where with that phrase any­more, in any study.

I be­lieve Wei_Lai would say that the first ap­proach, treat­ing eth­i­cal ax­ioms as “self-ev­i­dent truths” is prob­le­matic due to the fact that a lot of hy­po­thet­i­cal situ­a­tions (like my ex­am­ple be­fore) can cre­ate a lot of con­tra­dic­tions be­tween var­i­ous eth­i­cal ax­ioms (i.e. choos­ing be­tween tel­ling a lie and let­ting ter­ror­ists blow up the planet).

Not if those eth­i­cal ax­ioms ac­tu­ally are self-ev­i­dent truths. Then hy­po­thet­i­cal situ­a­tions (no mat­ter how un­com­fortable they make us) can’t dis­rupt them. But we might, on the ba­sis of these situ­a­tions, con­clude that we don’t have any self-ev­i­dent moral ax­ioms. But, as you neatly ar­gue, we don’t have any self-ev­i­dent math­e­mat­i­cal ax­ioms ei­ther.

• Thanks for tak­ing the time to read and re­spond to the ar­ti­cle, and for the cri­tique; you are cor­rect in that I am not well-versed in Greek philos­o­phy. With that be­ing said, al­low me to try to ex­pand my frame­work to ex­plain what I’m try­ing to get at:

• Scien­tists, un­like math­e­mat­i­ci­ans, don’t always frame their ar­gu­ments in terms of pure logic (i.e. If A and B, then C). How­ever, I be­lieve that the work that comes from them can be treated as log­i­cal state­ments.

Ex­am­ple: “I think that heat is trans­ferred be­tween two ob­jects via some sort of mat­ter that I will call ‘phlo­gis­ton’. If my hy­poth­e­sis is true, than an ob­ject will lose mass as it cools down.” 10 days later: “I have weighed an ob­ject when it was hot, and I weighed it when it was cold. The ob­ject did not lose any mass. There­fore, my hy­poth­e­sis is wrong”.

In log­i­cal terms: Let’s call the The­ory of Phlo­gis­ton “A”, and let’s call the act of mea­sur­ing a loss of mass with a loss of heat “C”.

1. If A, then C.

2. Phys­i­cal ev­i­dence is obtained

3. If Not C, then Not A.

Essen­tially, the sci­en­tific method in­volves the cre­ation of a hy­poth­e­sis “A”, and a log­i­cal con­se­quence of that hy­poth­e­sis, “If A then C”. Then phys­i­cal ev­i­dence is pre­sented in fa­vor of, or against “C”. If C is dis­proven, then A is dis­proven.

This is what I mean when I say that hy­pothe­ses are “ax­ioms”, and phys­i­cal ex­per­i­ments are “con­clu­sions”.

• In re­sponse to this state­ment:

Socrates, in the Pla­tonic di­alogues, is un­will­ing to take the law of non-con­tra­dic­tion as an ax­iom. There just aren’t any ax­ioms in So­cratic philos­o­phy, just dis­cus­sions. No proofs, just con­ver­sa­tions. Plato (and cer­tainly not Socrates) doesn’t have doc­trines, and Plato is to­tally and in­ten­tion­ally mer­ciless with peo­ple who try to find Pla­tonic doc­trines.

“No proofs, just con­ver­sa­tions”. In the frame­work that I’m work­ing in, ev­ery sin­gle state­ment is ei­ther a premise or a con­clu­sion. In ad­di­tion, ev­ery sin­gle state­ment is ei­ther a “truth” (that we are to be­lieve im­me­di­ately), a “propo­si­tion” (that we are to en­ter­tain the log­i­cal im­pli­ca­tions of), or part of a “hy­poth­e­sis/​im­pli­ca­tion” pair (that we are sup­pose to be­lieve with a level of skep­ti­cism un­til an ex­per­i­ment ver­ifies it or dis­proves it). I be­lieve that ev­ery sin­gle state­ment that has ever been made in any field of study falls into one of those 3 cat­e­gories, and I’m say­ing that we need to dis­cuss which cat­e­gory we need to place state­ments that are in the field of ethics.

In the field of philos­o­phy, from my limited knowl­edge, I think that these dis­cus­sions lead to con­clu­sions that we need to be­lieve as “truth”, whether or not they are sup­ported by ev­i­dence (i.e. John Rawl’s “Origi­nal Po­si­tion”).

• This is what I mean when I say that hy­pothe­ses are “ax­ioms”, and phys­i­cal ex­per­i­ments are “con­clu­sions”.

I see. You’re right that philoso­phers pretty much never do any­thing like that. Ex­cept ex­per­i­men­tal philoso­phers, but thus far most of that stuff is just ter­rible.

“In the frame­work that I’m work­ing in...”

That’s a good frame­work with with to ap­proach any philo­soph­i­cal text, in­clud­ing and es­pe­cially the Pla­tonic di­alogues. I just wanted to stress the fact that the di­alogues aren’t trea­tises pre­sented in a funny way. You’re sup­posed to ar­gue with Socrates, against him, yell at his in­ter­locu­tors, try to patch up the ar­gu­ments with premises of your own. It’s very differ­ent from, say, Aris­to­tle or Kant or what­ever, where its a guy pre­sent­ing a the­ory.

In the field of philos­o­phy, from my limited knowl­edge, I think that these dis­cus­sions lead to con­clu­sions that we need to be­lieve as “truth”

Would you mind if I go on for a bit? I have thoughts on this, but I don’t quite know how to pre­sent them briefly. Any­way:

Stu­dents of Physics should go into a Physics class room or book with an open mind. They should be ready to learn new things about the world, of­ten sur­pris­ing things (rel­a­tive to their naive im­pres­sions) and should of­ten try to check their prej­u­dices at the door. None of us are born know­ing physics. It’s some­thing we have to go out and learn.

Philos­o­phy isn’t like that. The right at­ti­tude walk­ing into a philos­o­phy class­room is ir­ri­ta­tion. It is an in­her­ently an­noy­ing sub­ject, and its prac­ti­tion­ers are even worse. You can’t learn philos­o­phy, and you can’t be­come an ex­pert at it. You can’t even be­come good at it. Be­ing a philoso­pher is no ac­com­plish­ment what­so­ever. You can just do philos­o­phy, and any­one can do it. In­tel­li­gence is good, but it can be a hin­drance too, same with ed­u­ca­tion.

Do­ing philos­o­phy means ask­ing ques­tions about things to which you re­ally ought to already know the an­swers, like the differ­ence be­tween right and wrong, whether or not you’re in con­trol of your ac­tions, what change is, what ex­ist­ing is, etc. Philos­o­phy is about ask­ing ques­tions to which we ought to have the an­swers, but don’t.

We do philos­o­phy by talk­ing to each other. If that means run­ning an ex­per­i­ment, good. If that means just ar­gu­ing, fine. There’s no method, no stan­dards, and no body of knowl­edge, un­less you say there is, and then con­vince some­one, and then there is un­til some­one con­vinces you oth­er­wise.

Scien­tists and math­e­mat­i­ci­ans don’t hate philos­o­phy. They tend to love philoso­phers, or at least the older ones do. Young sci­en­tists and math­e­mat­i­ci­ans do hate philoso­phers, and with good rea­son: part of be­ing a young sci­en­tist or math­e­mat­i­cian is de­vel­op­ing a re­fined men­tal self-dis­ci­pline, and that means turn­ing your back on any froo-froo hand wavy BS and get­ting down to work. Philos­o­phy is the most hate­ful thing in the world when you’re try­ing to be wrong as lit­tle as pos­si­ble. But once that dis­ci­pline is in place, and peo­ple are con­fi­dent in their abil­ity to sort out good ar­gu­ments from bad ones, facts from spec­u­la­tion, philos­o­phy starts to look like fun.

• The sec­ond part of your post is ter­rific. :)

• In the frame­work that I’m work­ing in, ev­ery sin­gle state­ment is ei­ther a premise or a con­clu­sion. In ad­di­tion, ev­ery sin­gle state­ment is ei­ther a “truth” (that we are to be­lieve im­me­di­ately), a “propo­si­tion” (that we are to en­ter­tain the log­i­cal im­pli­ca­tions of), or part of a “hy­poth­e­sis/​im­pli­ca­tion” pair (that we are sup­pose to be­lieve with a level of skep­ti­cism un­til an ex­per­i­ment ver­ifies it or dis­proves it)

But there is a mini-premise, in­fer­ence and mini-con­clu­sion in­side ev­ery “hy­poth­e­sis-im­pli­ca­tion pair”.

• In the field of philos­o­phy, from my limited knowl­edge, I think that these dis­cus­sions lead to con­clu­sions that we need to be­lieve as “truth”, whether or not they are sup­ported by ev­i­dence (i.e. John Rawl’s “Origi­nal Po­si­tion”).

I’m cu­ri­ous as to why you refer­enced Rawl’s work in this con­text. It’s not ap­par­ent to me how Jus­tice as Fair­ness is rele­vant here.

• I refer­enced him be­cause I re­call that he comes to a very strong con­clu­sion- that a moral so­ciety should have agreed-upon laws based on the premise of the “origi­nal po­si­tion”. He was the first philoso­pher that came to mind when I was try­ing to think of ex­am­ples of a hard state­ment that is nei­ther a “propo­si­tion” to be ex­plored, nor the con­clu­sion from an ob­serv­able fact.

• I mean, I’m pretty sure his con­clu­sion is a “propo­si­tion.” It has premises, and I could con­struct it log­i­cally if you wanted.

In fact, I don’t un­der­stand his po­si­tion to be “that a moral so­ciety should have agreed-upon laws” at all, but rather his use of the origi­nal po­si­tion is an at­tempt to iso­late and dis­cover the prin­ci­ples of dis­tribu­tive jus­tice, and that’s re­ally his bot­tom line.

• Thank you for an awe­some read. :)

• sci­ence uses log­i­cal rules of in­fer­ence. Does sci­ence take them as self-ev­i­dent? Or does it test them? And can it test them with­out as­sum­ing them?

• (whisper: Wei Lai should be Wei Dai)

• Nope. Even if one grants ob­jec­tive mean­ing to a unique in­ter­per­sonal ag­gre­gate of suffer­ing (and I don’t), it’s just wrong.

Some­times you want peo­ple to suffer. For ex­am­ple, if one fel­low caused all the suffer­ing of the rest, mov­ing him to less suffer­ing than ev­ery­one else would be a move to a worse uni­verse.

EDIT: I didn’t mean “you” to in­di­cate ev­ery­one. Some­times I want peo­ple to suffer, and think that in my hy­po­thet­i­cal, the ma­jor­ity of mankind would feel the same, and choose the same, if it were in their power.

• Some­times you want peo­ple to suffer. For ex­am­ple, if one fel­low caused all the suffer­ing of the rest, mov­ing him to less suffer­ing than ev­ery­one else would be a move to a worse uni­verse.

...be­cause do­ing so would cre­ate in­cen­tive to not cause suffer­ing to oth­ers. In the long run, that would re­sult in less uni­ver­sal suffer­ing over­all. Isn’t this cor­rect?

• No, that’s not my mo­ti­va­tion at all. That’s not my be­cause. It’s just vengeance on my part.

Even if one re­garded the de­sign of vengeance as an evolu­tion­ary adap­ta­tion, I don’t think that vengeance min­i­mizes suffer­ing, it pun­ishes in­frac­tions against val­ues.

At that level, it’s not about min­i­miz­ing suffer­ing ei­ther, it’s about evolu­tion­ary fit­ness.

• Yeah, I’m pretty sure I (and most LWers) don’t agree with you on that one, at least in the way you phrased it.

• You think they’d pre­fer that the guy that caused ev­ery­one else in the uni­verse to suffer didn’t suffer him­self?

• Here’s an old Eliezer quote on this:

4.5.2: Doesn’t that screw up the whole con­cept of moral re­spon­si­bil­ity?

Hon­estly? Well, yeah. Mo­ral re­spon­si­bil­ity doesn’t ex­ist as a phys­i­cal ob­ject. Mo­ral re­spon­si­bil­ity—the idea that choos­ing evil causes you to de­serve pain—is fun­da­men­tally a hu­man idea that we’ve all adopted for con­ve­nience’s sake. (23).

The truth is, there is ab­solutely noth­ing you can do that will make you de­serve pain. Sad­dam Hus­sein doesn’t de­serve so much as a stubbed toe. Pain is never a good thing, no mat­ter who it hap­pens to, even Adolf Hitler. Pain is bad; if it’s ul­ti­mately mean­ingful, it’s al­most cer­tainly as a nega­tive goal. Noth­ing any hu­man be­ing can do will flip that sign from nega­tive to pos­i­tive.

So why do we throw peo­ple in jail? To dis­cour­age crime. Choos­ing evil doesn’t make a per­son de­serve any­thing wrong, but it makes ver tar­getable, so that if some­thing bad has to hap­pen to some­one, it may as well hap­pen to ver. Adolf Hitler, for ex­am­ple, is so tar­getable that we could shoot him on the off-chance that it would save some­one a stubbed toe. There’s never a point where we can morally take plea­sure in some­one else’s pain. But hu­man so­ciety doesn’t re­quire ha­tred to func­tion—just law.

Be­sides which, my mind feels a lot cleaner now that I’ve to­tally re­nounced all ha­tred.

It’s pretty hard to ar­gue about this if our moral in­tu­itions dis­agree. But at least, you should know that most peo­ple on LW dis­agree with you on this in­tu­ition.

EDIT: As ArisKat­saris points out, I don’t ac­tu­ally have any source for the “most peo­ple on LW dis­agree with you” bit. I’ve always thought that not want­ing harm to come to any­one as an in­stru­men­tal value was a pretty ob­vi­ous, stan­dard part of util­i­tar­i­anism, and 62% of LWers are con­se­quen­tial­ist, ac­cord­ing to the 2012 sur­vey. The post “Policy De­bates Should Not Ap­pear One Sided” is fairly highly re­garded, and it es­poses a re­lated view, that peo­ple don’t de­serve harm for their stu­pidity.

Also, what those peo­ple would pre­fer isn’t nesse­car­ily what our moral sys­tem should pre­fer- hu­mans are petty and short-sighted.

• I’ve always thought that not want­ing harm to come to any­one as an in­stru­men­tal value was a pretty ob­vi­ous, stan­dard part of util­i­tar­i­anism, and 62% of LWers are con­se­quen­tial­ist, ac­cord­ing to the 2012 sur­vey.

What do you mean by “util­i­tar­i­anism”? The word has two differ­ent com­mon mean­ings around here: any type of con­se­quen­tial­ism, and the spe­cific type of con­se­quen­tial­ism that uses “to­tal hap­piness” as a util­ity func­tion. This sen­tence ap­pears to be de­signed to con­fuse the two mean­ings.

The post “Policy De­bates Should Not Ap­pear One Sided” is fairly highly re­garded, and it es­poses a re­lated view, that peo­ple don’t de­serve harm for their stu­pidity.

That is most definitely not the main point of that post.

• What do you mean by “util­i­tar­i­anism”? The word has two differ­ent com­mon mean­ings around here: any type of con­se­quen­tial­ism, and the spe­cific type of con­se­quen­tial­ism that uses “to­tal hap­piness” as a util­ity func­tion. This sen­tence ap­pears to be de­signed to con­fuse the two mean­ings.

Yeah, my mis­take. I’d never run across any other ver­sions of con­se­quen­tial­ism apart from util­i­tar­i­anism (ex­cept for Clippy, of course). I sup­pose car­ing only for your­self might count? But do you se­ri­ously think that the ma­jor­ity of those con­se­quen­tial­ists aren’t util­i­tar­ian?

• Well, even Eliezer’s ver­sion of con­se­quen­tial­ism isn’t sim­ple util­i­tar­i­anism for starters.

• It’s a kind of util­i­tar­i­anism. I’m in­clud­ing act util­i­tar­i­anism and de­sire util­i­tar­i­anism and prefer­ence util­i­tar­i­anism and what­ever in util­i­tar­i­anism.

• Ok, what is your defi­ni­tion of “util­i­tar­i­anism”?

• But at least, you should know that most peo­ple on LW dis­agree with you on this in­tu­ition.

[cita­tion needed]

• I ed­ited my com­ment to in­clude a tiny bit more ev­i­dence.

• Thank you, that’s a good start.

Yes, I had con­cluded that EY was anti re­tri­bu­tion. Hadn’t con­cluded that he had car­ried the day on that point.

Mo­ral re­spon­si­bil­ity—the idea that choos­ing evil causes you to de­serve pain—is fun­da­men­tally a hu­man idea that we’ve all adopted for con­ve­nience’s sake. (23).

I don’t think vengeance and re­tri­bu­tion are “ideas” that peo­ple had to come up with—they’re cen­tral moral mo­ti­va­tions. “A so­cial prefer­ence for which we pun­ish vi­o­la­tors” gets at 80% of what moral­ity is about.

Some may dis­agree about the in­tu­ition, but I’d note that even EY had to “re­nounce” all ha­tred, which im­plies to me that he had the im­pulse for ha­tred (re­tri­bu­tion, in this con­text) in the first place.

This seems like it has mak­ings of an in­ter­est­ing poll ques­tion.

• This seems like it has mak­ings of an in­ter­est­ing poll ques­tion.

I agree. Let’s do that. You’re con­se­quen­tial­ist, right?

I’d phrase my opinion as “I have ter­mi­nal value for peo­ple not suffer­ing, in­clud­ing peo­ple who have done some­thing wrong. I ac­knowl­edge that some­times caus­ing suffer­ing might have in­stru­men­tal value, such as im­pris­on­ment for crimes.”

How do you phrase yours? If I were to guess, it would be “I have a ter­mi­nal value which says that peo­ple who have caused suffer­ing should suffer them­selves.”

• I’d sug­gest the fol­low­ing two phras­ings:

• I place ter­mi­nal value to re­tri­bu­tion (in­flict­ing suffer­ing on the causers of suffer­ing), at least for some of the most egre­gious cases.

• I do not place ter­mi­nal value to re­tri­bu­tion, not even for the most egre­gious cases (e.g. mass mur­der­ers). I ac­knowl­edge that some­times it may have in­stru­men­tal value.

Per­haps also add a third choice:

• I think I place ter­mi­nal value to re­tri­bu­tion, but I would pre­fer it if I could self-mod­ify so that I wouldn’t.

• I would, all else be­ing equal. Suffer­ing is bad.

• The prac­tice of moral philos­o­phy doesn’t much re­sem­ble the prac­tice of math­e­mat­ics. Mainly be­cause in moral philos­o­phy we don’t know ex­actly what we’re talk­ing about when we talk about moral­ity. In math­e­mat­ics, par­tic­u­larly since the 20th cen­tury, we can even­tu­ally pre­cisely spec­ify what we mean by a math­e­mat­i­cal ob­ject, in terms of sets.

“Mo­ral­ity is logic” means that when we talk about moral­ity we are talk­ing about a math­e­mat­i­cal ob­ject. The fact that the only place in our mind the refer­ence to this ob­ject is stored is our in­tu­ition is what makes moral philos­o­phy so difficult and non-log­icy. In prac­tice you can’t write down a com­plete syn­tac­tic de­scrip­tion of moral­ity, so in gen­eral¹ nei­ther can you write syn­tac­tic proofs of the­o­rems about moral­ity. This is not to say that such de­scrip­tions or proofs do not ex­ist!

In prac­tice moral philos­o­phy pro­ceeds by a kind of prob­a­bil­is­tic rea­son­ing, which might be analo­gized to the think­ing that leads one to con­jec­ture that P≠NP, ex­cept with even less rigor. I’d ex­pect that things like the or­der of moral ar­gu­ments mat­ter­ing come down to fram­ing effects and other bi­ases which are always in­volved re­gard­less of the sub­ject, but don’t show up in math­e­mat­ics so much be­cause proofs leave lit­tle wig­gle room.

¹ Of course, you may be able to write proofs that only use sim­ple prop­er­ties that you can be fairly sure hold of moral­ity with­out know­ing its full de­scrip­tion, but such prop­er­ties are usu­ally ei­ther quite bor­ing or not widely agreed upon or don’t lead to in­ter­est­ing proofs due to be­ing too spe­cific. eg. “It’s wrong to kill some­one with­out their per­mis­sion when there’s noth­ing to be gained by it.”

• “Mo­ral­ity is logic” means that when we talk about moral­ity we are talk­ing about a math­e­mat­i­cal ob­ject.

How does one go about defin­ing this math­e­mat­i­cal ob­ject, in prin­ci­ple? Sup­pose you were a su­per­in­tel­li­gence and could sur­mount any kind of tech­ni­cal difficulty, and you wanted to define a hu­man’s moral­ity pre­cisely as a math­e­mat­i­cal ob­ject, how would you do it?

• I don’t re­ally know the an­swer to that ques­tion.

In prin­ci­ple, you start with a hu­man brain, and ex­tract from it some­how a de­scrip­tion of what it means when it says “moral­ity”. Pre­sum­ably in­volv­ing some kind of anal­y­sis of what would make the hu­man say “that’s good!” or “that’s bad!”, and/​or of what com­pu­ta­tional pro­cesses in­side the brain are in­volved in de­cid­ing whether to say “good” or “bad”. The out­put is, in the­ory, a func­tion map­ping things to how much they match “good” or “bad” in your hu­man’s lan­guage.

The ‘sim­ple’ solu­tion, of just simu­lat­ing what your hu­man would say af­ter be­ing ex­posed to ev­ery pos­si­ble moral ar­gu­ment, runs into trou­ble with what ex­actly con­sti­tutes an ar­gu­ment—if an UFAI can hack your brain into do­ing ter­rible things just by talk­ing to you, clearly not all ver­bal en­gage­ment can be al­lowed—and also more mun­dane is­sues like our simu­lated hu­man go­ing in­sane from all this talk­ing.

• Sup­pose the “sim­ple” solu­tion doesn’t have the prob­lems you men­tion. Some­how we get our hands on a hu­man that doesn’t have se­cu­rity holes and can’t go in­sane. I still don’t think it works.

Let’s say you are try­ing to do some prob­a­bil­is­tic rea­son­ing about the math­e­mat­i­cal ob­ject “foo­bar” and the defi­ni­tion of it you’re given is “foo­bar is what X would say about ‘foo­bar’ af­ter be­ing ex­posed to ev­ery pos­si­ble ar­gu­ment con­cern­ing ‘foo­bar’”, where X is an al­gorith­mic de­scrip­tion of your­self. Well, as soon as you re­al­ize that X is ac­tu­ally a simu­la­tion of you, you can con­clude that you can say any­thing about ‘foo­bar’ and be right. So why bother do­ing any more prob­a­bil­is­tic rea­son­ing? Just say any­thing, or noth­ing. What kind of prob­a­bil­is­tic rea­son­ing can do you be­yond that, even if you wanted to?

• I think you’re col­laps­ing some lev­els here, but it’s mak­ing my head hurt to think about it, hav­ing the defi­ni­tion-de­river and the sub­ject be the same per­son.

Mak­ing this con­crete: let ‘foo­bar’ re­fer to the set {1, 2, 3} in a shared lan­guage used by us and our sub­ject, Alice. Alice would agree that it is true that “foo­bar = what X would say about ‘foo­bar’ af­ter be­ing ex­posed to ev­ery pos­si­ble ar­gu­ment con­cern­ing ‘foo­bar’” where X is some al­gorith­mic de­scrip­tion of Alice. She would say some­thing like “foo­bar = {1, 2, 3}, X would say {1, 2, 3}, {1, 2, 3} = {1, 2, 3} so this all checks out.”

Clearly then, any pro­ce­dure that cor­rectly de­ter­mines what X would say about ‘foo­bar’ should re­sult in the cor­rect defi­ni­tion of foo­bar, namely {1, 2, 3}. This is what the­o­ret­i­cally lets our “sim­ple” solu­tion work.

How­ever, Alice would not agree that “what X would say about ‘foo­bar’ af­ter be­ing ex­posed to ev­ery pos­si­ble ar­gu­ment con­cern­ing ‘foo­bar’” is a cor­rect defi­ni­tion of ‘foo­bar’. The is­sue is that this defi­ni­tion has the wrong prop­er­ties when we con­sider coun­ter­fac­tu­als con­cern­ing X. It is in fact the case that foo­bar is {1, 2, 3}, and fur­ther that ‘foo­bar’ means {1, 2, 3} in our cur­rent lan­guage, as stipu­lated at the be­gin­ning of this thought ex­per­i­ment. If-coun­ter­fac­tu­ally X would say ‘{4, 5, 6}‘, foo­bar is still {1, 2, 3}, be­cause what we mean by ‘foo­bar’ is {1, 2, 3} and {1, 2, 3} is {1, 2, 3} re­gard­less of what X says.

Hav­ing writ­ten that, I now think I can re­turn to your ques­tion. The an­swer is that firstly, by re­plac­ing the true defi­ni­tion “foo­bar = {1, 2, 3}” with “foo­bar is what X would say about ‘foo­bar’ af­ter be­ing ex­posed to ev­ery pos­si­ble ar­gu­ment con­cern­ing ‘foo­bar’” in the sub­ject’s mind, you have just deleted the only refer­ence to foo­bar that ac­tu­ally ex­ists in the thought ex­per­i­ment. The sub­ject has to rea­son about ‘foo­bar’ us­ing their built in defi­ni­tion, since that is the only thing that ac­tu­ally points di­rectly to the tar­get ob­ject.

Se­condly, as de­scribed above “foo­bar is what X would say about ‘foo­bar’ af­ter be­ing ex­posed to ev­ery pos­si­ble ar­gu­ment con­cern­ing ‘foo­bar’” is an in­ac­cu­rate defi­ni­tion of foo­bar when con­sid­er­ing coun­ter­fac­tu­als con­cern­ing what X would say about foo­bar. Which is ex­actly what you are do­ing when rea­son­ing that “if-coun­ter­fac­tu­ally I say {4, 5, 6} about foo­bar, then what X would say about ‘foo­bar’ is {4, 5, 6}, so {4, 5, 6} is cor­rect.”

Which is to say that, anal­o­gis­ing, the con­tents of our sub­ject’s head is a poin­ter (in the pro­gram­ming sense) to the ob­ject it­self, while “what X would say about ‘foo­bar’ af­ter be­ing ex­posed to ev­ery pos­si­ble ar­gu­ment con­cern­ing ‘foo­bar’” is a poin­ter to the first poin­ter. You can derefer­ence it, and get the right an­swer, but you can’t just sub­sti­tute it in for the first poin­ter. That gives you noth­ing but a poin­ter refer­ring to it­self.

ETA: Dear god, this turned into a long post. Sorry! I don’t think I can shorten it with­out mak­ing it worse though.

• Right, so my point is that if your the­ory (that moral rea­son­ing is prob­a­bil­is­tic rea­son­ing about some math­e­mat­i­cal ob­ject) is to be cor­rect, we need a defi­ni­tion of moral­ity as a math­e­mat­i­cal ob­ject which isn’t “what X says af­ter con­sid­er­ing all pos­si­ble moral ar­gu­ments”. So what could it be then? What defi­ni­tion Y can we give, such that it makes sense to say “when we rea­son about moral­ity, we are re­ally do­ing prob­a­bil­is­tic rea­son­ing about the math­e­mat­i­cal ob­ject Y”?

Se­condly, un­til we have a can­di­date defi­ni­tion Y at hand, we can’t show that moral rea­son­ing re­ally does cor­re­spond to prob­a­bil­is­tic log­i­cal rea­son­ing about Y. (And we’d also have to first un­der­stand what “prob­a­bil­is­tic log­i­cal rea­son­ing” is.) So, at this point, how can we be con­fi­dent that moral rea­son­ing does cor­re­spond to prob­a­bil­is­tic log­i­cal rea­son­ing about any­thing math­e­mat­i­cal, and isn’t just some sort of ran­dom walk or some sort of rea­son­ing that’s differ­ent from prob­a­bil­is­tic log­i­cal rea­son­ing?

• Right, so my point is that if your the­ory (that moral rea­son­ing is prob­a­bil­is­tic rea­son­ing about some math­e­mat­i­cal ob­ject) is to be cor­rect, we need a defi­ni­tion of moral­ity as a math­e­mat­i­cal ob­ject which isn’t “what X says af­ter con­sid­er­ing all pos­si­ble moral ar­gu­ments”. So what could it be then? What defi­ni­tion Y can we give, such that it makes sense to say “when we rea­son about moral­ity, we are re­ally do­ing prob­a­bil­is­tic rea­son­ing about the math­e­mat­i­cal ob­ject Y”?

Un­for­tu­nately I doubt I can give you a short di­rect defi­ni­tion of moral­ity. How­ever if such a math­e­mat­i­cal ob­ject ex­ists, “what X says af­ter con­sid­er­ing all pos­si­ble moral ar­gu­ments” should be enough to pin it down (dis­re­gard­ing the caveats to do with our sub­ject go­ing in­sane, etc).

Se­condly, un­til we have a can­di­date defi­ni­tion Y at hand, we can’t show that moral rea­son­ing re­ally does cor­re­spond to prob­a­bil­is­tic log­i­cal rea­son­ing about Y. (And we’d also have to first un­der­stand what “prob­a­bil­is­tic log­i­cal rea­son­ing” is.) So, at this point, how can we be con­fi­dent that moral rea­son­ing does cor­re­spond to prob­a­bil­is­tic log­i­cal rea­son­ing about any­thing math­e­mat­i­cal, and isn’t just some sort of ran­dom walk or some sort of rea­son­ing that’s differ­ent from prob­a­bil­is­tic log­i­cal rea­son­ing?

Well, I think it safe to as­sume I mean some­thing by moral talk, oth­er­wise I wouldn’t care so much about whether things are right or wrong. I must be talk­ing about some­thing, be­cause that some­thing is wired into my de­ci­sion sys­tem. And I pre­sume this some­thing is math­e­mat­i­cal, be­cause (as­sum­ing I mean some­thing by “P is good”) you can take the set of all good things, and this set is the same in all coun­ter­fac­tu­als. Roughly speak­ing.

It is, of course, pos­si­ble that moral rea­son­ing isn’t ac­tu­ally any kind of valid rea­son­ing, but does amount to a “ran­dom walk” of some kind, where con­sid­er­ing an ar­gu­ment per­ma­nently changes your in­tu­ition in some non­de­ter­minis­tic way so that af­ter hear­ing the ar­gu­ment you’re not even talk­ing about the same thing you were be­fore hear­ing it. Which is wor­ry­ing.

Also it’s pos­si­ble that moral talk in par­tic­u­lar is mostly sig­nal­ling in­tended to dis­guise our true val­ues which are very similar but more self­ish. But that doesn’t make a lot of differ­ence since you can still cash out your val­ues as a math­e­mat­i­cal ob­ject of some sort.

• It is, of course, pos­si­ble that moral rea­son­ing isn’t ac­tu­ally any kind of valid rea­son­ing, but does amount to a “ran­dom walk” of some kind, where con­sid­er­ing an ar­gu­ment per­ma­nently changes your in­tu­ition in some non­de­ter­minis­tic way so that af­ter hear­ing the ar­gu­ment you’re not even talk­ing about the same thing you were be­fore hear­ing it. Which is wor­ry­ing.

Yes, ex­actly. This seems to me pretty likely to be the case for hu­mans. Even if it’s ac­tu­ally not the case, no­body has done the work to rule it out yet (has any­one even writ­ten a post mak­ing any kind of ar­gu­ment that it’s not the case?), so how do we know that it’s not the case? Doesn’t it seem to you that we might be do­ing some mo­ti­vated cog­ni­tion in or­der to jump to a com­fort­ing con­clu­sion?

• “what X says af­ter con­sid­er­ing all pos­si­ble moral ar­gu­ments”

I know you’re not ar­gu­ing for this but I can’t help not­ing the dis­crep­ancy be­tween the sim­plic­ity of the phrase “all pos­si­ble moral ar­gu­ments”, and what it would mean if it can be defined at all.

But then many things are “eas­ier said than done”.

• I think the term you are look­ing for is “for­mal” or “an alge­bra”, not “logic”.

• I like this post, and here is some ev­i­dence sup­port­ing your fear that some peo­ple may over-use the moral­ity=logic metaphor, i.e., copy too many an­ti­ci­pa­tions about how log­i­cal rea­son­ing works over to their an­ti­ci­pa­tions about how moral rea­son­ing works… The com­ment is already down­voted to −2, sug­gest­ing the com­mu­nity re­al­izes this (please don’t down­vote it fur­ther so as to over-pun­ish the au­thor), but the fact that some­one made it is ev­i­dence that your point here is valuable one.

http://​​less­wrong.com/​​lw/​​g0e/​​nar­ra­tive_self­i­mage_and_self­com­mu­ni­ca­tion/​​83ag

• You’re mischar­ac­ter­iz­ing the quote that your post replies to. EY claims that he is at­tempt­ing to com­pre­hend moral­ity as a log­i­cal, not a phys­i­cal thing, and he’s try­ing to con­vince read­ers to do the same. You’re ev­i­dently think­ing of moral­ity as a phys­i­cal thing, some­thing es­sen­tially de­rived from the ob­ser­va­tion of brains. You’re restat­ing the po­si­tion his post re­sponds to, with­out strength­en­ing it.

• The ar­gu­ment in that post seems in­co­her­ent to me. In the con­ven­tional nat­u­ral sci­ences, what is moral is the sub­ject mat­ter of biol­ogy. This em­ploys game the­ory and evolu­tion­ary the­ory (i.e. logic), but also con­sid­ers the laws of physics and the lo­cal state of the uni­verse to ex­plain ex­ist­ing moral sys­tems.

For in­stance, con­sider the ques­tion of whether it is wrong to drive on the left-hand side of the road. That isn’t logic, it de­pends on the lo­cal state of the uni­verse. Two ad­vanced su­per­in­tel­li­gences which had evolved in­de­pen­dently could eas­ily find them­selves in dis­agree­ment over such is­sues.

This is an ex­am­ple of spon­ta­neous sym­me­try break­ing. It is one of the fac­tors which ex­plains how ar­bi­trar­ily-ad­vanced agents can still dis­agree on what the right thing to do is.