Morality is Scary

[Wei Dai]

I’m wor­ried that many AI al­ign­ment re­searchers and other LWers have a view of how hu­man moral­ity works, that re­ally only ap­plies to a small frac­tion of all hu­mans (no­tably moral philoso­phers and them­selves). In this view, peo­ple know or at least sus­pect that they are con­fused about moral­ity, and are ea­ger or will­ing to ap­ply rea­son and de­liber­a­tion to find out what their real val­ues are, or to cor­rect their moral be­liefs. Here’s an ex­am­ple of some­one who fits this view:

I’ve writ­ten, in the past, about a “ghost” ver­sion of my­self — that is, one that can float free from my body; which travel any­where in all space and time, with un­limited time, en­ergy, and pa­tience; and which can also make changes to differ­ent vari­ables, and play for­ward/​rewind differ­ent coun­ter­fac­tual timelines (the ghost’s ac­tivity some­how doesn’t have any moral sig­nifi­cance).

I some­times treat such a ghost kind of like an ideal­ized self. It can see much that I can­not. It can see di­rectly what a small part of the world I truly am; what my ac­tions truly mean. The lives of oth­ers are real and vivid for it, even when hazy and out of mind for me. I trust such a per­spec­tive a lot. If the ghost would say “don’t,” I’d be in­clined to listen.

I’m cur­rently read­ing The Sta­tus Game by Will Storr (highly recom­mended BTW), and found in it the fol­low­ing de­scrip­tion of how moral­ity works in most peo­ple, which matches my own un­der­stand­ing of his­tory and my ob­ser­va­tions of hu­mans around me:

The moral re­al­ity we live in is a virtue game. We use our dis­plays of moral­ity to man­u­fac­ture sta­tus. It’s good that we do this. It’s func­tional. It’s why billion­aires fund libraries, uni­ver­sity schol­ar­ships and sci­en­tific en­deav­ours; it’s why a study of 11,672 or­gan dona­tions in the USA found only thirty-one were made anony­mously. It’s why we feel good when we com­mit moral acts and thoughts pri­vately and en­joy the ap­proval of our imag­i­nary au­di­ence. Virtue sta­tus is the bribe that nudges us into putting the in­ter­ests of other peo­ple – prin­ci­pally our co-play­ers – be­fore our own.

We treat moral be­liefs as if they’re uni­ver­sal and ab­solute: one study found peo­ple were more likely to be­lieve God could change phys­i­cal laws of the uni­verse than he could moral ‘facts’. Such facts can seem to be­long to the same cat­e­gory as ob­jects in na­ture, as if they could be ob­served un­der micro­scopes or proven by math­e­mat­i­cal for­mu­lae. If moral truth ex­ists any­where, it’s in our DNA: that an­cient game-play­ing cod­ing that evolved to nudge us into be­hav­ing co-op­er­a­tively in hunter-gath­erer groups. But these in­struc­tions – strive to ap­pear vir­tu­ous; priv­ilege your group over oth­ers – are few and vague and open to ri­o­tous differ­ences in in­ter­pre­ta­tion. All the rest is an act of shared imag­i­na­tion. It’s a dream we weave around a sta­tus game.

The dream shifts as we range across the con­ti­nents. For the Mala­gasy peo­ple in Mada­gas­car, it’s taboo to eat a blind hen, to dream about blood and to sleep fac­ing west­wards, as you’ll kick the sun­rise. Ado­les­cent boys of the Marind of South New Guinea are in­tro­duced to a cul­ture of ‘in­sti­tu­tion­al­ised sodomy’ in which they sleep in the men’s house and ab­sorb the sperm of their el­ders via anal cop­u­la­tion, mak­ing them stronger. Among the peo­ple of the Moose, teenage girls are ab­ducted and forced to have sex with a mar­ried man, an act for which, writes psy­chol­o­gist Pro­fes­sor David Buss, ‘all con­cerned – in­clud­ing the girl – judge that her par­ents giv­ing her to the man was a vir­tu­ous, gen­er­ous act of grat­i­tude’. As alien as these norms might seem, they’ll feel morally cor­rect to most who play by them. They’re part of the dream of re­al­ity in which they ex­ist, a dream that feels no less ob­vi­ous and true to them than ours does to us.

Such ‘facts’ also change across time. We don’t have to travel back far to dis­cover moral su­per­stars hold­ing moral views that would de­stroy them to­day. Fem­i­nist hero and birth con­trol cam­paigner Marie Stopes, who was voted Wo­man of the Millen­nium by the read­ers of The Guardian and hon­oured on spe­cial Royal Mail stamps in 2008, was an anti-Semite and eu­geni­cist who once wrote that ‘our race is weak­ened by an ap­pal­lingly high per­centage of un­fit weak­lings and dis­eased in­di­vi­d­u­als’ and that ‘it is the ur­gent duty of the com­mu­nity to make par­ent­hood im­pos­si­ble for those whose men­tal and phys­i­cal con­di­tions are such that there is well-nigh a cer­tainty that their offspring must be phys­i­cally and men­tally tainted’. Mean­while, Gandhi once ex­plained his ag­i­ta­tion against the Bri­tish thusly: ‘Ours is one con­tinual strug­gle against a degra­da­tion sought to be in­flicted upon us by the Euro­peans, who de­sire to de­grade us to the level of the raw Kaf­fir [black Afri­can] … whose sole am­bi­tion is to col­lect a cer­tain num­ber of cat­tle to buy a wife with and … pass his life in in­dolence and naked­ness.’ Such state­ments seem ob­vi­ously ap­pal­ling. But there’s about as much sense in blam­ing Gandhi for not shar­ing our mod­ern, Western views on race as there is in blam­ing the Vik­ings for not hav­ing Net­flix. Mo­ral ‘truths’ are acts of imag­i­na­tion. They’re ideas we play games with.

The dream feels so real. And yet it’s all con­jured up by the game-mak­ing brain. The world around our bod­ies is chaotic, con­fus­ing and mostly un­know­able. But the brain must make sense of it. It has to turn that bliz­zard of noise into a pre­cise, colour­ful and de­tailed world it can pre­dict and suc­cess­fully in­ter­act with, such that it gets what it wants. When the brain dis­cov­ers a game that seems to make sense of its felt re­al­ity and offer a path­way to re­wards, it can em­brace its rules and sym­bols with an ec­static fer­vour. The noise is silenced! The chaos is tamed! We’ve found our story and the heroic role we’re go­ing to play in it! We’ve learned the truth and the way – the mean­ing of life! It’s yams, it’s God, it’s money, it’s sav­ing the world from evil big pHARMa. It’s not like a re­li­gious ex­pe­rience, it is a re­li­gious ex­pe­rience. It’s how the writer Arthur Koestler felt as a young man in 1931, join­ing the Com­mu­nist Party:

‘To say that one had “seen the light” is a poor de­scrip­tion of the men­tal rap­ture which only the con­vert knows (re­gard­less of what faith he has been con­verted to). The new light seems to pour from all di­rec­tions across the skull; the whole uni­verse falls into pat­tern, like stray pieces of a jig­saw puz­zle as­sem­bled by one magic stroke. There is now an an­swer to ev­ery ques­tion, doubts and con­flicts are a mat­ter of the tor­tured past – a past already re­mote, when one lived in dis­mal ig­no­rance in the taste­less, colourless world of those who don’t know. Noth­ing hence­forth can dis­turb the con­vert’s in­ner peace and seren­ity – ex­cept the oc­ca­sional fear of los­ing faith again, los­ing thereby what alone makes life worth liv­ing, and fal­ling back into the outer dark­ness, where there is wailing and gnash­ing of teeth.’

I hope this helps fur­ther ex­plain why I think even solv­ing (some ver­sions of) the al­ign­ment prob­lem prob­a­bly won’t be enough to en­sure a fu­ture that’s free from as­tro­nom­i­cal waste or as­tro­nom­i­cal suffer­ing. A part of me is ac­tu­ally more scared of many fu­tures in which “al­ign­ment is solved”, than a fu­ture where biolog­i­cal life is sim­ply wiped out by a pa­per­clip max­i­mizer.

[Daniel Kokotajlo]

I think this post makes an im­por­tant point—or rather, raises a very im­por­tant ques­tion, with some vivid ex­am­ples to get you started. On the other hand, I feel like it doesn’t go fur­ther, and prob­a­bly should have—I wish it e.g. sketched a con­crete sce­nario in which the fu­ture is dystopian not be­cause we failed to make our AGIs “moral” but be­cause we suc­ceeded, or e.g. got a bit more for­mal and com­ple­mented the quotes with a toy model (in­spired by the quotes) of how moral de­liber­a­tion in a so­ciety might work, un­der post-AGI-al­ign­ment con­di­tions, and how that could sys­tem­at­i­cally lead to dystopia un­less we man­age to be fore­sight­ful and set up the so­cial con­di­tions just right.

I recom­mend not in­clud­ing this post, and in­stead in­clud­ing this one and Wei Dai’s ex­change in the com­ments.

[TekhneMakre]

> All the rest is an act of shared imag­i­na­tion. It’s a dream we weave around a sta­tus game.
> They’re part of the dream of re­al­ity in which they ex­ist, a dream that feels no less ob­vi­ous and true to them than ours does to us.
> Mo­ral ‘truths’ are acts of imag­i­na­tion. They’re ideas we play games with.

IDK, I feel like you could say the same sen­tences truth­fully about math, and if you “went with the over­all vibe” of them, you might be con­fused and mis­tak­enly think math was “ar­bi­trary” or “mean­ingless”, or doesn’t have a de­ter­mi­nate ten­dency, etc. Like, okay, if I say “one el­e­ment of moral progress is in­creas­ing uni­ver­sal­iz­abil­ity”, and you say “that’s just the thing your sta­tus co­hort as­signs high sta­tus”, I’m like, well, sure, but that doesn’t mean it doesn’t also have other in­ter­est­ing prop­er­ties, like be­ing a ten­dency across many differ­ent peo­ples; like be­ing cor­re­lated with the ex­tent to which they’re re­flect­ing, shar­ing in­for­ma­tion, and build­ing un­der­stand­ing; like re­sult­ing in re­duc­tion­ist-ma­te­ri­al­ist lo­cal out­comes that have more of ma­te­rial lo­cal things that peo­ple oth­er­wise gen­er­ally seem to like (e.g. not be­ing punched, hav­ing food, etc.); etc. It could be that moral­ity has ten­den­cies, but not with­out horme­sis and mu­tu­ally as­sured de­strub­tion and similar things that might be re­moved by al­igned AI.

[fourier]

Like, okay, if I say “one el­e­ment of moral progress is in­creas­ing uni­ver­sal­iz­abil­ity”, and you say “that’s just the thing your sta­tus co­hort as­signs high sta­tus”, I’m like, well, sure, but that doesn’t mean it doesn’t also have other in­ter­est­ing prop­er­ties, like be­ing a ten­dency across many differ­ent peo­ples; like be­ing cor­re­lated with the ex­tent to which they’re re­flect­ing, shar­ing in­for­ma­tion, and build­ing un­der­stand­ing; like re­sult­ing in re­duc­tion­ist-ma­te­ri­al­ist lo­cal out­comes that have more of ma­te­rial lo­cal things that peo­ple oth­er­wise gen­er­ally seem to like (e.g. not be­ing punched, hav­ing food, etc.);

“Mo­ral­ity” is to­tally un­like math­e­mat­ics where the rules can first be clearly defined, and we op­er­ate with that set of rules.

I be­lieve “in­creas­ing uni­ver­sal­iz­abil­ity” is a good ex­am­ple to prove OPs point. I don’t think it’s a com­mon be­lief among “many differ­ent peo­ples” in any mean­ingful sense. I don’t even re­ally un­der­stand what it en­tails. There may be a few nearly uni­ver­sal el­e­ments like “want­ing food”, but de­struc­tive as­pects are fun­da­men­tal to our lives so you can’t just re­move them with­out fun­da­men­tally al­ter­ing our na­ture as hu­man be­ings. Like a lot of peo­ple, I don’t mind be­ing punched a lit­tle as long as (me /​ my fam­ily /​ my group) wins and gains more re­sources. I re­ally want to see the peo­ple I hate be­ing harmed, and would sac­ri­fice a lot for it, that’s a very fun­da­men­tal as­pect of be­ing hu­man.

[TekhneMakre]

I re­ally want to see the peo­ple I hate be­ing harmed, and would sac­ri­fice a lot for it, that’s a very fun­da­men­tal as­pect of be­ing hu­man.

Are you pur­su­ing this to any great ex­tent? If so, re­mind me to stay away from you and avoid in­vest­ing in you.

[fourier]

Why are you per­son­ally at­tack­ing me for dis­cussing the topic at hand? I’m dis­cussing hu­man na­ture and giv­ing my­self as a counter-ex­am­ple, but I clearly meant that it ap­plies to ev­ery­one in differ­ent ways. I will avoid per­sonal ex­am­ples since some peo­ple have a hard time un­der­stand­ing. I be­lieve you are iron­i­cally prov­ing my point by sig­nal­ing against me based on my be­liefs which you dis­like.

[TekhneMakre]

At­tack­ing you? I said I don’t want to be around you and don’t want to in­vest in you. I said it with a touch of snark (“re­mind me”).

> I clearly meant that it ap­plies to ev­ery­one in differ­ent ways

Not clear to me. I don’t think ev­ery­one “would sac­ri­fice a lot” to “see the peo­ple [they] hate be­ing harmed”. I wouldn’t. I think be­hav­ing that way is in­ad­vis­able for you and harm­ful to oth­ers, and will tend to make you a bad in­vest­ment op­por­tu­nity.

[TekhneMakre]

“Mo­ral­ity” is to­tally un­like math­e­mat­ics where the rules can first be clearly defined, and we op­er­ate with that set of rules.

By that de­scrip­tion, math­e­mat­ics is fairly un­like math­e­mat­ics.

I don’t even re­ally un­der­stand what it en­tails.

It en­tails that be­hav­ior that peo­ple con­sider moral, tends to­wards hav­ing the prop­erty that if ev­ery­one be­haved like that, things would be good. Rule of law, equal­ity be­fore the law, Rawlsian veil of ig­no­rance, stare de­ci­sis, equal­ity of op­por­tu­nity, the golden rule, liberty, etc. Gen­er­ally, norms that are sym­met­ric across space, time, con­text, and per­son. (Not say­ing we ac­tu­ally have these things, or that “most peo­ple” ex­plic­itly think these things are good, just that peo­ple tend to up­date in fa­vor of these things.)

[fourier]

It en­tails that be­hav­ior that peo­ple con­sider moral, tends to­wards hav­ing the prop­erty that if ev­ery­one be­haved like that, things would be good

This is just cir­cu­lar. What is “good”?

Rule of law, equal­ity be­fore the law, Rawlsian veil of ig­no­rance, stare de­ci­sis, equal­ity of op­por­tu­nity, the golden rule, liberty, etc. Gen­er­ally, norms that are sym­met­ric across space, time, con­text, and per­son. (Not say­ing we ac­tu­ally have these things, or that “most peo­ple” ex­plic­itly think these things are good, just that peo­ple tend to up­date in fa­vor of these things.)

Ev­i­dence that “most peo­ple” up­date in fa­vor of these things? It seems like a very cur­rent west­ern moral­ity cen­tric view, and you could prob­a­bly get peo­ple to up­date in the op­po­site di­rec­tion (and they did, many times in his­tory).

[TekhneMakre]

>Ev­i­dence that “most peo­ple” up­date in fa­vor of these things? It seems like a very cur­rent west­ern moral­ity cen­tric view,

Yeah, I think you’re right that it’s bi­ased to­wards Western. I think you can gen­er­ate the ob­vi­ous ex­am­ples (e.g. law sys­tems de­vel­op­ing; e.g. var­i­ous rev­olu­tions in the name of liberty and equal­ity and against tyranny), and I’m not in­ter­ested enough right now to come up with more com­pre­hen­sive treat­ment of the ev­i­dence, and I’m not su­per con­fi­dent. It could be in­ter­est­ing to see how this plays out in places where these ten­den­cies seem least pre­sent. Is China such a place? (What do most peo­ple liv­ing in China re­ally think of non-liberty, non-Rawlsi­anism, etc.?)

[Sammy Martin]

The above sen­tences, if taken (as you do) as claims about hu­man moral psy­chol­ogy rather than nor­ma­tive ethics, are com­pat­i­ble with full-on moral re­al­ism. I.e. ev­ery­one’s moral at­ti­tudes are pushed around by sta­tus con­cerns, luck­ily we ended up in a com­mu­nity that ties sta­tus to look­ing for long-run im­pli­ca­tions of your be­liefs and mak­ing sure they’re co­her­ent, and so with­out hav­ing fun­da­men­tally differ­ent mo­ti­va­tions to any other hu­man be­ing we were bet­ter able to be mo­ti­vated by ac­tual moral facts.

I know the OP is try­ing to say loudly and re­peat­edly that this isn’t the case be­cause ‘ev­ery­one else thought that as well, don’t you know?’ with lots of vivid ex­am­ples, but if that’s the only ar­gu­ment it seems like mod­esty episte­mol­ogy—i.e. “most peo­ple who said the thing you said were wrong, and also said that they weren’t like all those other peo­ple who were wrong in the past for all these spe­cific rea­sons, so you should be­lieve you’re wrong too”.

I think a lot of this thread con­fuses moral psy­chol­ogy with nor­ma­tive ethics—most util­i­tar­i­ans know and un­der­stand that they aren’t solely mo­ti­vated by moral con­cerns, and are also mo­ti­vated by lots of other things. They know they don’t morally en­dorse those mo­ti­va­tions in them­selves, but don’t do any­thing about it, and don’t thereby change their moral views.

If Peter Singer goes and buys a coffee, it’s no ar­gu­ment at all to say “aha, by re­vealed prefer­ences, you must not re­ally think util­i­tar­i­anism is true, or you’d have given the money away!” That doesn’t show that when he does donate money, he’s un­mo­ti­vated by moral con­cerns.

Prob­a­bly even this ‘pure’ mo­ti­va­tion to act morally in cases where em­pa­thy isn’t much of an is­sue is it­self made up of e.g. a de­sire not to be seen be­liev­ing self-con­tra­dic­tory things, cog­ni­tive dis­so­nance, ba­sic em­pa­thy and so on. But so what? If the emo­tional in­cen­tives work to mo­ti­vate peo­ple to form more co­her­ent moral views, it’s the re­li­a­bil­ity of the pro­cess of form­ing the views that mat­ter, not the mo­ti­va­tion. I’m sure you could tell a similar story about the mo­ti­va­tions that drive math­e­mat­i­ci­ans to check their proofs are valid.

[Vanessa Kosoy]

You sound like you’re posit­ing the ex­is­tence of two type of peo­ple: type I peo­ple who have moral­ity based on “rea­son” and type II peo­ple who have moral­ity based on the “sta­tus game”. In re­al­ity, ev­ery­one’s nearly ev­ery­one’s moral­ity is based on some­thing like the sta­tus game (see also: 1 2 3). It’s just that EAs and moral philoso­phers are play­ing the game in a tribe which awards sta­tus differ­ently.

The true in­trin­sic val­ues of most peo­ple do place a weight on the hap­piness of other peo­ple (that’s roughly what we call “em­pa­thy”), but this weight is very un­equally dis­tributed.

There are definitely thorny ques­tions re­gard­ing the best way to ag­gre­gate the val­ues of differ­ent peo­ple in TAI. But, I think that given a rea­son­able solu­tion, a lower bound on the fu­ture is imag­in­ing that the AI will build a pri­vate utopia for ev­ery per­son, as iso­lated from the other “utopias” as that per­son wants it to be. Prob­a­bly some peo­ple’s “utopias” will not be great, viewed in util­i­tar­ian terms. But, I still pre­fer that over pa­per­clips (by far). And, I sus­pect that most peo­ple do (even if they protest it in or­der to play the game).

[Wei Dai]

It’s just that EAs and moral philoso­phers are play­ing the game in a tribe which awards sta­tus differ­ently.

Sure, I’ve said as much in re­cent com­ments, in­clud­ing this one. ETA: Re­lated to this, I’m wor­ried about AI dis­rupt­ing “our” sta­tus game in an un­pre­dictable and pos­si­bly dan­ger­ous way. E.g., what will hap­pen when ev­ery­one uses AI ad­vi­sors to help them play sta­tus games, in­clud­ing the sta­tus game of moral philos­o­phy?

The true in­trin­sic val­ues of most peo­ple do place a weight on the hap­piness of other peo­ple (that’s roughly what we call “emapthy”), but this weight is very un­equally dis­tributed.

What do you mean by “true in­trin­sic val­ues”? (I couldn’t find any pre­vi­ous us­age of this term by you.) How do you pro­pose find­ing peo­ple’s true in­trin­sic val­ues?

Th­ese weights, if low enough rel­a­tive to other “val­ues”, haven’t pre­vented peo­ple from com­mit­ting atroc­i­ties on each other in the name of moral­ity.

There are definitely thorny ques­tions re­gard­ing the best way to ag­gre­gate the val­ues of differ­ent peo­ple in TAI. But, I think that given a rea­son­able solu­tion, a lower bound on the fu­ture is imag­in­ing that the AI will build a pri­vate utopia for ev­ery per­son, as iso­lated from the other “utopias” as that per­son wants it to be.

This im­plies solv­ing a ver­sion of the al­ign­ment prob­lem that in­cludes rea­son­able value ag­gre­ga­tion be­tween differ­ent peo­ple (or be­tween AIs al­igned to differ­ent peo­ple), but at least some re­searchers don’t seem to con­sider that part of “al­ign­ment”.

Given that play­ing sta­tus games and sta­tus com­pe­ti­tion be­tween groups/​tribes/​sta­tus games con­sti­tute a huge part of peo­ple’s lives, I’m not sure how pri­vate utopias that are very iso­lated from each other would work. Also, I’m not sure if your solu­tion would pre­vent peo­ple from in­stan­ti­at­ing simu­la­tions of per­ceived en­e­mies /​ “evil peo­ple” in their utopias and pun­ish­ing them, or just simu­lat­ing a bunch of low sta­tus peo­ple to lord over.

Prob­a­bly some peo­ple’s “utopias” will not be great, viewed in util­i­tar­ian terms. But, I still pre­fer that over pa­per­clips (by far).

I con­cede that a util­i­tar­ian would prob­a­bly find al­most all “al­igned” fu­tures bet­ter than pa­per­clips. Per­haps I should have clar­ified that by “parts of me” be­ing more scared, I meant the self­ish and NU-lean­ing parts. The util­i­tar­ian part of me is just wor­ried about the po­ten­tial waste caused by many or most “utopias” be­ing very sub­op­ti­mal in terms of value cre­ated per unit of re­source con­sumed.

[Vanessa Kosoy]

What do you mean by “true in­trin­sic val­ues”? (I couldn’t find any pre­vi­ous us­age of this term by you.) How do you pro­pose find­ing peo­ple’s true in­trin­sic val­ues?

I mean the val­ues rel­a­tive to which a per­son seems most like a ra­tio­nal agent, ar­guably for­mal­iz­able along these lines.

Th­ese weights, if low enough rel­a­tive to other “val­ues”, haven’t pre­vented peo­ple from com­mit­ting atroc­i­ties on each other in the name of moral­ity.

Yes.

This im­plies solv­ing a ver­sion of the al­ign­ment prob­lem that in­cludes rea­son­able value ag­gre­ga­tion be­tween differ­ent peo­ple (or be­tween AIs al­igned to differ­ent peo­ple), but at least some re­searchers don’t seem to con­sider that part of “al­ign­ment”.

Yes. I do think multi-user al­ign­ment is an im­por­tant prob­lem (and oc­ca­sion­ally spend some time think­ing about it), it just seems rea­son­able to solve sin­gle user al­ign­ment first. An­drew Critch is an ex­am­ple of a per­son who seems to be con­cerned about this.

Given that play­ing sta­tus games and sta­tus com­pe­ti­tion be­tween groups/​tribes/​sta­tus games con­sti­tute a huge part of peo­ple’s lives, I’m not sure how pri­vate utopias that are very iso­lated from each other would work.

I meant that each pri­vate utopia can con­tain any num­ber of peo­ple cre­ated by the AI, in ad­di­tion to its “cus­tomer”. Ofc groups that can agree on a com­mon utopia can band to­gether as well.

Also, I’m not sure if your solu­tion would pre­vent peo­ple from in­stan­ti­at­ing simu­la­tions of per­ceived en­e­mies /​ “evil peo­ple” in their utopias and pun­ish­ing them, or just simu­lat­ing a bunch of low sta­tus peo­ple to lord over.

They are pre­vented from simu­lat­ing other pre-ex­ist­ing peo­ple with­out their con­sent, but can simu­late a bunch of low sta­tus peo­ple to lord over. Yes, this can be bad. Yes, I still pre­fer this (as­sum­ing my own pri­vate utopia) over pa­per­clips. And, like I said, this is just a rel­a­tively easy to imag­ine lower bound, not nec­es­sar­ily the true op­ti­mum.

Per­haps I should have clar­ified that by “parts of me” be­ing more scared, I meant the self­ish and NU-lean­ing parts.

The self­ish part, at least, doesn’t have any rea­son to be scared as long as you are a “cus­tomer”.

[Wei Dai]

They are pre­vented from simu­lat­ing other pre-ex­ist­ing peo­ple with­out their consent

Why do you think this will be the re­sult of the value ag­gre­ga­tion (or a lower bound on how good the ag­gre­ga­tion will be)? For ex­am­ple, if there is a big block of peo­ple who all want to simu­late per­son X in or­der to pun­ish that per­son, and only X and a few other peo­ple ob­ject, why won’t the value ag­gre­ga­tion be “no­body pre-ex­ist­ing ex­cept X (and Y and Z etc.) can be simu­lated”?

[Vanessa Kosoy]

Given some as­sump­tions about the do­mains of the util­ity func­tions, it is pos­si­ble to do bet­ter than what I de­scribed in the pre­vi­ous com­ment. Let $X_i$ be the space of pos­si­ble ex­pe­rience his­to­ries^[1] of user $i$ and $Y$ the space of ev­ery­thing else the util­ity func­tions de­pend on (things that no­body can ob­serve di­rectly). Sup­pose that the do­main of the util­ity func­tions is $Z:={\prod }_i{X}_i\times Y$. Then, we can define the “denos­ing^[2] op­er­a­tor” $D_i:C\left(Z\right)\to C\left(Z\right)$ for user $i$ by

$$\left(D_{i}u\right)(x_i,x_{-i},y):=\underset{x^{\prime }\in {\prod }_{j\ne i}{X}_j}{max}u(x_i,x^{\prime },y)$$

Here, $x_i$ is the ar­gu­ment of $u$ that ranges in $X_i$, $x_{-i}$ are the ar­gu­ments that range in $X_j$ for $j\ne i$ and $y$ is the ar­gu­ment that ranges in $Y$.

That is, $D_i$ mod­ifies a util­ity func­tion by hav­ing it “imag­ine” that the ex­pe­riences of all users other than $i$ have been op­ti­mized, for the ex­pe­riences of user $i$ and the un­ob­serv­ables held con­stant.

Let $u_i:Z\to R$ be the util­ity func­tion of user $i$, and $d^0\in R^n$ the ini­tial dis­agree­ment point (ev­ery­one dy­ing), where $n$ is the num­ber of users. We then perform co­op­er­a­tive bar­gain­ing on the denosed util­ity func­tions $D_i{u}_i$ with dis­agree­ment point $d^0$, pro­duc­ing some out­come ${\mu }_0\in \Delta \left(Z\right)$. Define $d^1\in R^n$ by $d_i^1:=E_{\mu }\left[u_i\right]$. Now we do an­other co­op­er­a­tive bar­gain­ing with $d^1$ as the dis­agree­ment point and the origi­nal util­ity func­tions $u_i$. This gives us the fi­nal out­come ${\mu }_1$.

Among other benefits, there is now much less need to re­move out­liers. Per­haps, in­stead of re­mov­ing them we still want to miti­gate them by ap­ply­ing “am­plified denos­ing” to them which also re­moves the de­pen­dence on $Y$.

For this pro­ce­dure, there is a much bet­ter case that the lower bound will be met.

---

1. ↩︎

   In the stan­dard RL for­mal­ism this is the space of ac­tion-ob­ser­va­tion se­quences $(A\times O{)}^{\omega }$.

2. ↩︎

   From the ex­pres­sion “nosy prefer­ences”, see e.g. here.

[Joe Collman]

This is very in­ter­est­ing (and “denos­ing op­er­a­tor” is delight­ful).

Some thoughts:

If I un­der­stand cor­rectly, I think there can still be a prob­lem where user $i$ wants an ex­pe­rience his­tory such that part of the his­tory is iso­mor­phic to a simu­la­tion of user $j$ suffer­ing ($i$ wants to fully ex­pe­rience $j$ suffer­ing in ev­ery de­tail).

Here a fixed $x_i$ may en­tail some fixed $x_j$ for (some copy of) some $j$.

It seems the above ap­proach can’t then avoid leav­ing one of $i$ or $j$ badly off:
If $i$ is per­mit­ted to freely de­ter­mine the ex­pe­rience of the em­bed­ded $j$ copy, the dis­agree­ment point in the sec­ond bar­gain­ing will bake this in: $j$ may be hor­rified to see that $i$ wants to ex­pe­rience its copy suffer, but will be pow­er­less to stop it (if $i$ won’t budge in the bar­gain­ing).

Con­versely, if the em­bed­ded $j$ is treated as a user which $i$ will imag­ine is ex­actly to $i$’s lik­ing, but who ac­tu­ally gets what $j$ wants, then the se­lected ${\mu }_0$ will be hor­rible for $i$ (e.g. per­haps $i$ wants to fully ex­pe­rience Hitler suffer­ing, and in­stead gets to fully ex­pe­rience Hitler’s wildest fan­tasies be­ing re­al­ized).

I don’t think it’s pos­si­ble to do any­thing like denos­ing to avoid this.

It may seem like this isn’t a prac­ti­cal prob­lem, since we could rea­son­ably dis­al­low such em­bed­ding. How­ever, I think that’s still tricky since there’s a less ex­otic ver­sion of the is­sue: my ex­pe­riences likely already are a col­lec­tion of sub­agents’ ex­pe­riences. Pre­sum­ably my max­imi­sa­tion over $x_{joe}$ is per­mit­ted to de­ter­mine all the $x_{subjoe}$.

It’s hard to see how you draw a prin­ci­pled line here: the ideal fu­ture for most peo­ple may eas­ily be tran­shu­man­ist to the point where to­day’s users are to­mor­row’s sub­per­son­al­ities (and be­yond).

A case that may have to be ruled out sep­a­rately is where $i$ wants to be­come a suffer­ing $j$. Depend­ing on what I con­sider ‘me’, I might be en­tirely fine with it if ‘I’ wake up to­mor­row as suffer­ing $j$ (if I’m done liv­ing and think $j$ de­serves to suffer).
Or per­haps I want to clone my­self ${10}^{10}$ times, and then have all copies con­vert them­selves to suffer­ing $j$s af­ter a while. [in gen­eral, it seems there has to be some mechanism to dis­tribute re­sources rea­son­ably—but it’s not en­tirely clear what that should be]

[Vanessa Kosoy]

I think that a rigor­ous treat­ment of such is­sues will re­quire some var­i­ant of IB phys­i­cal­ism (in which the mono­ton­ic­ity prob­lem has been solved, some­how). I am cau­tiously op­ti­mistic that a denos­ing op­er­a­tor ex­ists there which dodges these prob­lems. This op­er­a­tor will de­clare both the man­i­fest­ing and eval­u­a­tion of the source codes of other users to be “out of scope” for a given user. Hence, a prefer­ence of $i$ to ob­serve the suffer­ing of $j$ would be “satis­fied” by ob­serv­ing nearly any­thing, since the max­i­miza­tion can in­ter­pret any­thing as a simu­la­tion of $j$.

The “sub­joe” prob­lem is differ­ent: it is ir­rele­vant be­cause “sub­joe” is not a user, only Joe is a user. All the tran­shu­man­ist magic that hap­pens later doesn’t change this. Users are peo­ple liv­ing dur­ing the AI launch, and only them. The sta­tus of any fu­ture (trans/​post)hu­mans is de­ter­mined en­tirely ac­cord­ing to the util­ity func­tions of users. Why? For two rea­sons: (i) the AI can only have ac­cess and sta­ble poin­t­ers to ex­ist­ing peo­ple (ii) we only need the buy-in of ex­ist­ing peo­ple to launch the AI. If ex­ist­ing peo­ple want fu­ture peo­ple to be treated well, then they have noth­ing to worry about since this prefer­ence is part of the ex­ist­ing peo­ple’s util­ity func­tions.

[Joe Collman]

Ah—that’s cool if IB phys­i­cal­ism might ad­dress this kind of thing (still on my to-read list).

Agreed that the sub­joe thing isn’t di­rectly a prob­lem. My worry is mainly whether it’s harder to rule out $i$ ex­pe­rienc­ing a simu­la­tion of $x_{subj-suffering}$, since sub$j$ isn’t a user. How­ever, if you can avoid the suffer­ing $j$s by limit­ing ac­cess to in­for­ma­tion, the same should pre­sum­ably work for rele­vant sub-$j$s.

If ex­ist­ing peo­ple want fu­ture peo­ple to be treated well, then they have noth­ing to worry about since this prefer­ence is part of the ex­ist­ing peo­ple’s util­ity func­tions.

This isn’t so clear (to me at least) if:

1. Most, but not all cur­rent users want fu­ture peo­ple to be treated well.

2. Part of be­ing “treated well” in­cludes be­ing in­volved in an on­go­ing bar­gain­ing pro­cess which de­cides the AI’s/​fu­ture’s tra­jec­tory.

For in­stance, sup­pose ini­tially 90% of peo­ple would like to have an iter­ated bar­gain­ing pro­cess that in­cludes fu­ture (trans/​post)hu­mans as users, once they ex­ist. The other 10% are only will­ing to ac­cept such a situ­a­tion if they main­tain their bar­gain­ing power in fu­ture iter­a­tions (by what­ever mechanism).

If you iter­ate this pro­cess, the bar­gain­ing pro­cess ends up dom­i­nated by users who won’t re­lin­quish any power to fu­ture users. 90% of ini­tial users might pre­fer drift over lock-in, but we get lock-in re­gard­less (the dis­agree­ment point also amount­ing to lock-in).

Un­less I’m con­fus­ing my­self, this kind of thing seems like a prob­lem. (not in terms of reach­ing some non-ter­rible lower bound, but in terms of re­al­is­ing po­ten­tial)
Wher­ever there’s this kind of asym­me­try/​degra­da­tion over bar­gain­ing iter­a­tions, I think there’s an ar­gu­ment for build­ing in a way to avoid it from the start—since any­thing short of 100% just limits to 0 over time. [it’s by no means clear that we do want to make fu­ture peo­ple users on an equal foot­ing to to­day’s peo­ple; it just seems to me that we have to do it at step zero or not at all]

[Vanessa Kosoy]

Ah—that’s cool if IB phys­i­cal­ism might ad­dress this kind of thing

I ad­mit that at this stage it’s un­clear be­cause phys­i­cal­ism brings in the mono­ton­ic­ity prin­ci­ple that cre­ates big­ger prob­lems than what we dis­cuss here. But maybe some var­i­ant can work.

For in­stance, sup­pose ini­tially 90% of peo­ple would like to have an iter­ated bar­gain­ing pro­cess that in­cludes fu­ture (trans/​post)hu­mans as users, once they ex­ist. The other 10% are only will­ing to ac­cept such a situ­a­tion if they main­tain their bar­gain­ing power in fu­ture iter­a­tions (by what­ever mechanism).

Roughly speak­ing, in this case the 10% pre­serve their 10% of the power for­ever. I think it’s fine be­cause I want the buy-in of this 10% and the cost seems ac­cept­able to me. I’m also not sure there is any vi­able al­ter­na­tive which doesn’t have even big­ger prob­lems.

[Joe Collman]

Sure, I’m not sure there’s a vi­able al­ter­na­tive ei­ther. This kind of ap­proach seems promis­ing—but I want to bet­ter un­der­stand any down­sides.

My worry wasn’t about the ini­tial 10%, but about the pos­si­bil­ity of the pro­cess be­ing iter­ated such that you end up with al­most all bar­gain­ing power in the hands of power-keep­ers.

In ret­ro­spect, this is prob­a­bly silly: if there’s a des­ignable-by-us mechanism that bet­ter achieves what we want, the first bar­gain­ing iter­a­tion should find it. If not, then what I’m ges­tur­ing at must ei­ther be in­co­her­ent, or not en­dorsed by the 10% - so hard-cod­ing it into the ini­tial mechanism wouldn’t get the buy-in of the 10% to the ex­tent that they un­der­stood the mechanism.

In the end, I think my con­cern is that we won’t get buy-in from a large ma­jor­ity of users:
In or­der to ac­com­mo­date some pro­por­tion with odd moral views it seems likely you’ll be throw­ing away huge amounts of ex­pected value in oth­ers’ views—if I’m cor­rectly in­ter­pret­ing your pro­posal (please cor­rect me if I’m con­fused).

Is this where you’d want to ap­ply am­plified denos­ing?
So, rather than fil­ter­ing out the un­de­sir­able $i$, for these $i$ you use:

$\left(D_{i}u\right)(x_i,x_{-i},y):={\underset{x^{\prime }\in {\prod }_{j\ne i}{X}_j,y^{\prime }\in Y}{max}}^{}u(x_i,x^{\prime },y^{\prime })$ [i.e. ig­nor­ing y and imag­in­ing it’s op­ti­mal]

How­ever, it’s not clear to me how we’d de­cide who gets strong denos­ing (clearly not ev­ery­one, or we don’t pick a $y$). E.g. if you strong-denose any­one who’s too will­ing to al­low bar­gain­ing failure [ev­ery­one dies] you might end up fil­ter­ing out al­tru­ists who worry about suffer­ing risks.
Does that make sense?

[Vanessa Kosoy]

My worry wasn’t about the ini­tial 10%, but about the pos­si­bil­ity of the pro­cess be­ing iter­ated such that you end up with al­most all bar­gain­ing power in the hands of power-keep­ers.

I’m not sure what you mean here, but also the pro­cess is not iter­ated: the ini­tial bar­gain­ing is de­cid­ing the out­come once and for all. At least that’s the math­e­mat­i­cal ideal we’re ap­prox­i­mat­ing.

In the end, I think my con­cern is that we won’t get buy-in from a large ma­jor­ity of users: In or­der to ac­com­mo­date some pro­por­tion with odd moral views it seems likely you’ll be throw­ing away huge amounts of ex­pected value in oth­ers’ views

I don’t think so? The bar­gain­ing sys­tem does ad­van­tage large groups over small groups.

In prac­tice, I think that for the most part peo­ple don’t care much about what hap­pens “far” from them (for some defi­ni­tion of “far”, not phys­i­cal dis­tance) so giv­ing them pri­vate utopias is close to op­ti­mal from each in­di­vi­d­ual per­spec­tive. Although it’s true they might pre­tend to care more than they do for the usual rea­sons, if they’re think­ing in “far-mode”.

I would cer­tainly be very con­cerned about any sys­tem that gives even more power to ma­jor­ity views. For ex­am­ple, what if the ma­jor­ity of peo­ple are dis­gusted by gay sex and pre­fer it not the hap­pen any­where? I would rather ac­cept things I dis­ap­prove of hap­pen­ing far away from me than al­low other peo­ple to con­trol my own life.

Ofc the sys­tem also man­dates win-win ex­changes. For ex­am­ple, if Alice’s and Bob’s pri­vate utopias each con­tain some­thing strongly un­palat­able to the other but not strongly im­por­tant to the re­spec­tive cus­tomer, the bar­gain­ing out­come will re­move both un­palat­able things.

E.g. if you strong-denose any­one who’s too will­ing to al­low bar­gain­ing failure [ev­ery­one dies] you might end up fil­ter­ing out al­tru­ists who worry about suffer­ing risks.

I’m fine with strong-denos­ing nega­tive ut­li­tar­i­anists who would truly stick to their guns about nega­tive util­i­tar­i­anism (but I also don’t think there are many).

[Joe Collman]

Ah, I was just be­ing an idiot on the bar­gain­ing sys­tem w.r.t. small num­bers of peo­ple be­ing able to hold it to ran­som. Oops. Agreed that more ma­jor­ity power isn’t de­sir­able.
[re iter­a­tion, I only meant that the bar­gain­ing could be­come iter­ated if the ini­tial bar­gain­ing re­sult were to de­cide upon iter­a­tion (to in­clude more fu­ture users). I now don’t think this is par­tic­u­larly sig­nifi­cant.]

I think my re­main­ing un­cer­tainty (/​con­fu­sion) is all re­lated to the is­sue I first men­tioned (em­bed­ded copy ex­pe­riences). It strikes me that some­thing like this can also hap­pen where minds grow/​merge/​over­lap.

This op­er­a­tor will de­clare both the man­i­fest­ing and eval­u­a­tion of the source codes of other users to be “out of scope” for a given user. Hence, a prefer­ence of $i$ to ob­serve the suffer­ing of $j$ would be “satis­fied” by ob­serv­ing nearly any­thing, since the max­i­miza­tion can in­ter­pret any­thing as a simu­la­tion of $j$.

Does this avoid the prob­lem if $i$’s prefer­ences use in­di­rec­tion? It seems to me that a ro­bust poin­ter to $j$ may be enough: that with a ro­bust poin­ter it may be pos­si­ble to im­plic­itly re­quire some­thing like source-code-ac­cess with­out ex­plic­itly refer­enc­ing it. E.g. where $i$ has a prefer­ence to “ex­pe­rience $j$ suffer­ing in cir­cum­stances where there’s strong ev­i­dence it’s ac­tu­ally $j$ suffer­ing, given that these cir­cum­stances were the out­come of this bar­gain­ing pro­cess”.

If $i$ can’t ro­bustly spec­ify things like this, then I’d guess there’d be sig­nifi­cant trou­ble in spec­i­fy­ing quite a few (mu­tu­ally) de­sir­able situ­a­tions in­volv­ing other users too. IIUC, this would only be any prob­lem for the denosed bar­gain­ing to find a good $d^1$: for the sec­ond bar­gain­ing on the true util­ity func­tions there’s no need to put any­thing “out of scope” (right?), so win-wins are eas­ily achieved.

[Vanessa Kosoy]

I’m imag­in­ing co­op­er­a­tive bar­gain­ing be­tween all users, where the dis­agree­ment point is ev­ery­one dy­ing^[1][2] (this is a nat­u­ral choice as­sum­ing that if we don’t build al­igned TAI we get pa­per­clips). This guaran­tees that ev­ery user will re­ceive an out­come that’s at least not worse than death.

With Nash bar­gain­ing, we can still get is­sues for (in)fa­mous peo­ple that mil­lions of peo­ple want to do un­pleas­ant things to. Their out­come will be bet­ter than death, but maybe worse than in my claimed “lower bound”.

With Kalai-Smorod­in­sky bar­gain­ing things look bet­ter, since es­sen­tially we’re max­i­miz­ing a min­i­mum over all users. This should ad­mit my lower bound, un­less it is some­how dis­rupted by enor­mous asym­me­tries in the max­i­mal pay­offs of differ­ent users.

In ei­ther case, we might need to do some kind of out­lier fil­ter­ing: if e.g. liter­ally ev­ery per­son on Earth is a user, then maybe some of them are ut­terly in­sane in ways that cause the Pareto fron­tier to col­lapse.

[EDIT: see im­proved solu­tion]

Bar­gain­ing as­sumes we can ac­cess the util­ity func­tion. In re­al­ity, even if we solve the value learn­ing prob­lem in the sin­gle user case, once you go to the multi-user case it be­comes a mechanism de­sign prob­lem: users have in­cen­tives to lie /​ mis­rep­re­sent their util­ity func­tions. A perfect solu­tion might be im­pos­si­ble, but I pro­posed miti­gat­ing this by as­sign­ing each user a vir­tual “AI lawyer” that pro­vides op­ti­mal in­put on their be­half into the bar­gain­ing sys­tem. In this case they at least have no in­cen­tive to lie to the lawyer, and the out­come will not be skewed in fa­vor of users who are bet­ter in this game, but we don’t get the op­ti­mal bar­gain­ing solu­tion ei­ther.

All of this as­sumes the TAI is based on some kind of value learn­ing. If the first-stage TAI is based on some­thing else, the prob­lem might be­come eas­ier or harder. Easier be­cause the first-stage TAI will pro­duce bet­ter solu­tions to the multi-user prob­lem for the sec­ond-stage TAI. Harder be­cause it can al­low the small group of peo­ple con­trol­ling it to im­pose their own prefer­ences.

For IDA-of-imi­ta­tion, de­moc­ra­ti­za­tion seems like a hard prob­lem be­cause the mechanism by which IDA-of-imi­ta­tion solves AI risk is pre­cisely by em­pow­er­ing a small group of peo­ple over ev­ery­one else (since the source of AI risk comes from other peo­ple launch­ing un­al­igned TAI). Ad­ding trans­parency can en­tirely un­der­mine safety.

For quan­tilized de­bate, adding trans­parency opens us to an at­tack vec­tor where the AI ma­nipu­lates pub­lic opinion. This sig­nifi­cantly low­ers the op­ti­miza­tion pres­sure bar for ma­nipu­la­tion, com­pared to ma­nipu­lat­ing the (care­fully se­lected) judges, which might un­der­mine the key as­sump­tion that effec­tive dishon­est strate­gies are harder to find than effec­tive hon­est strate­gies.

---

1. ↩︎

   This can be for­mal­ized by liter­ally hav­ing the AI con­sider the pos­si­bil­ity of op­ti­miz­ing for some un­al­igned util­ity func­tion. This is a weird and risky ap­proach but it works to 1st ap­prox­i­ma­tion.

2. ↩︎

   An al­ter­na­tive choice of dis­agree­ment point is max­i­miz­ing the util­ity of a ran­domly cho­sen user. This has ad­van­tages and dis­ad­van­tages.

[Wei Dai]

Bar­gain­ing as­sumes we can ac­cess the util­ity func­tion. In re­al­ity, even if we solve the value learn­ing prob­lem in the sin­gle user case, once you go to the multi-user case it be­comes a mechanism de­sign prob­lem: users have in­cen­tives to lie /​ mis­rep­re­sent their util­ity func­tions. A perfect solu­tion might be im­pos­si­ble, but I pro­posed miti­gat­ing this by as­sign­ing each user a vir­tual “AI lawyer” that pro­vides op­ti­mal in­put on their be­half into the bar­gain­ing sys­tem. In this case they at least have no in­cen­tive to lie to the lawyer, and the out­come will not be skewed in fa­vor of users who are bet­ter in this game, but we don’t get the op­ti­mal bar­gain­ing solu­tion ei­ther.

As­sum­ing each lawyer has the same in­cen­tive to lie as its client, it has an in­cen­tive to mis­rep­re­sent that some prefer­able-to-death out­comes are “worse-than-death” (in or­der to force those out­comes out of the set of “fea­si­ble agree­ments” in hope of get­ting a more preferred out­come as the ac­tual out­come), and this at equil­ibrium is bal­anced by the marginal in­crease in the prob­a­bil­ity of get­ting “ev­ery­one dies” as the out­come (due to fea­si­ble agree­ments be­com­ing a null set) caused by the lie. So the prob­a­bil­ity of “ev­ery­one dies” in this game has to be non-zero.

(It’s the same kind of prob­lem as in the AI race or tragedy of com­mons: peo­ple not tak­ing into ac­count the full so­cial costs of their ac­tions as they reach for pri­vate benefits.)

Of course in ac­tu­al­ity ev­ery­one dy­ing may not be a re­al­is­tic con­se­quence of failure to reach agree­ment, but if the real con­se­quence is bet­ter than that, and the AI lawyers know this, they would be more will­ing to lie since the per­ceived down­side of ly­ing would be smaller, so you end up with a higher chance of no agree­ment.

[Vanessa Kosoy]

Yes, it’s not a very satis­fac­tory solu­tion. Some al­ter­na­tive/​com­ple­men­tary solu­tions:

• Some­how use non-trans­for­ma­tive AI to do my mind up­load­ing, and then have the TAI to learn by in­spect­ing the up­loads. Would be great for sin­gle-user al­ign­ment as well.

• Some­how use non-trans­for­ma­tive AI to cre­ate perfect lie de­tec­tors, and use this to en­force hon­esty in the mechanism. (But, is it pos­si­ble to de­tect self-de­cep­tion?)

• Have the TAI learn from past data which wasn’t af­fected by the in­cen­tives cre­ated by the TAI. (But, is there enough in­for­ma­tion there?)

• Shape the TAI’s prior about hu­man val­ues in or­der to rule out at least the most blatant lies.

• Some clever mechanism de­sign I haven’t thought of. The prob­lem with this is, most mechanism de­signs rely on money and money that doesn’t seem ap­pli­ca­ble, whereas when you don’t have money there are many im­pos­si­bil­ity the­o­rems.

[Joe Collman]

In ei­ther case, we might need to do some kind of out­lier fil­ter­ing: if e.g. liter­ally ev­ery per­son on Earth is a user, then maybe some of them are ut­terly in­sane in ways that cause the Pareto fron­tier to col­lapse.

This seems near guaran­teed to me: a non-zero amount of peo­ple will be that crazy (in our terms), so fil­ter­ing will be nec­es­sary.

Then I’m cu­ri­ous about how we draw the line on out­lier fil­ter­ing. What fil­ter­ing rule do we use? I don’t yet see a good prin­ci­pled rule (e.g. if we want to throw out peo­ple who’d col­lapse agree­ment to the dis­agree­ment point, there’s more than one way to do that).

[deleted]

[Joe Collman]

Maybe crazy be­havi­our cor­re­lates with less intelligence

Depend­ing what we mean by ‘crazy’ I think that’s un­likely—par­tic­u­larly when what we care about here are highly un­usual moral stances. I’d see in­tel­li­gence as a mul­ti­plier, rather than some­thing which points you in the ‘right’ di­rec­tion. Out­liers will be at both ex­tremes of in­tel­li­gence—and I think you’ll get a much wider moral va­ri­ety on the high end.

For in­stance, I don’t think you’ll find many low-in­tel­li­gence anti­na­tal­ists—and here I mean the stronger, non-ob­vi­ous claim: not sim­ply that most peo­ple call­ing them­selves anti­na­tal­ists, or ad­vo­cat­ing for anti­na­tal­ism will have fairly high in­tel­li­gence, but rather that most peo­ple with such a moral stance (per­haps not ar­tic­u­lated) will have fairly high in­tel­li­gence.

Gen­er­ally, I think there are many weird moral stances you might think your way into that you’d be highly un­likely to find ‘nat­u­rally’ (through e.g. ab­sorp­tion of cul­tural norms).
I’d also ex­pect cre­ativity to pos­i­tively cor­re­late with out­lier moral­ities. Minds that ha­bit­u­ally throw to­gether seven dis­parate con­cepts will find cra­zier no­tions than those which don’t get be­yond three.

[deleted]

[Joe Collman]

First, I think we want to be think­ing in terms of [per­sonal moral­ity we’d re­flec­tively en­dorse] rather than [all the base, weird, con­flict­ing… drivers of be­havi­our that hap­pen to be in our heads].

There are things most of us would wish to change about our­selves if we could. There’s no sense in bak­ing them in for all eter­nity (or bar­gain­ing on their be­half), just be­cause they hap­pen to form part of what drives us now. [though one does have to be a bit care­ful here, since it’s easy to miss the up­side of qual­ities we re­gard as flaws]

With this in mind, re­flec­tively en­dorsed anti­na­tal­ism re­ally is a prob­lem: yes, some peo­ple will en­dorse sac­ri­fic­ing ev­ery­thing just to get to a world where there’s no suffer­ing (be­cause there are no peo­ple).

Note that the kinds of bar­gain­ing ap­proach Vanessa is ad­vo­cat­ing are aimed at guaran­tee­ing a lower bound for ev­ery­one (who’s not pre-filtered out) - so you only need to in­clude one per­son with a par­tic­u­larly weird view to fail to reach a sen­si­ble bar­gain. [though her most re­cent ver­sion should avoid this]

[Wei Dai]

Yes, I still pre­fer this (as­sum­ing my own pri­vate utopia) over pa­per­clips.

For a util­i­tar­ian, this doesn’t mean much. What’s much more im­por­tant is some­thing like, “How close is this out­come to an ac­tual (global) utopia (e.g., with op­ti­mized util­itro­n­ium filling the uni­verse), on a lin­ear scale?” For ex­am­ple, my rough ex­pec­ta­tion (with­out hav­ing thought about it much) is that your “lower bound” out­come is about mid­way be­tween pa­per­clips and ac­tual utopia on a log­a­r­ith­mic scale. In one sense, this is much bet­ter than pa­per­clips, but in an­other sense (i.e., on the lin­ear scale), it’s al­most in­dis­t­in­guish­able from pa­per­clips, and a util­i­tar­ian would only care about the lat­ter and there­fore be nearly as dis­ap­pointed by that out­come as pa­per­clips.

[Vanessa Kosoy]

I want to add a lit­tle to my stance on util­i­tar­i­anism. A util­i­tar­ian su­per­in­tel­li­gence would prob­a­bly kill me and ev­ery­one I love, be­cause we are made of atoms that could be used for minds that are more he­do­nic^[1][2][3]. Given a choice be­tween pa­per­clips and util­i­tar­i­anism, I would still choose util­i­tar­i­anism. But, if there was a util­i­tar­ian TAI pro­ject along with a half-de­cent chance to do some­thing bet­ter (by my lights), I would ac­tively op­pose the util­i­tar­ian pro­ject. From my per­spec­tive, such a pro­ject is es­sen­tially en­emy com­bat­ants.

---

1. ↩︎

   One way to avoid it is by mod­ify­ing util­i­tar­i­anism to only place weight on cur­rently ex­ist­ing peo­ple. But this is already not that far from my co­op­er­a­tive bar­gain­ing pro­posal (al­though still in­fe­rior to it, IMO).

2. ↩︎

   Another way to avoid it is by pos­tu­lat­ing some very strong penalty on death (i.e. dis­con­ti­nu­ity of per­son­al­ity). But this is not triv­ial to do, es­pe­cially with­out cre­at­ing other prob­lems. More­over, from my per­spec­tive this kind of thing is hacks try­ing to work around the core is­sue, namely that I am not a util­i­tar­ian (along with the vast ma­jor­ity of peo­ple).

3. ↩︎

   A pos­si­ble coun­ter­ar­gu­ment is, maybe the su­per­he­do­nic fu­ture minds would be sad to con­tem­plate our mur­der. But, this seems too weak to change the out­come, even as­sum­ing that this ver­sion of util­i­tar­i­anism man­dates minds who would want to know the truth and care about it, and that this prefer­ence is counted to­wards “util­ity”.

[Wei Dai]

A util­i­tar­ian su­per­in­tel­li­gence would prob­a­bly kill me and ev­ery­one I love, be­cause we are made of atoms that could be used for minds that are more hedonic

This seems like a rea­son­able con­cern about some types of he­do­nic util­i­tar­i­anism. To be clear, I’m not aware of any for­mu­la­tion of util­i­tar­i­anism that doesn’t have se­ri­ous is­sues, and I’m also not aware of any for­mu­la­tion of any moral­ity that doesn’t have se­ri­ous is­sues.

But, if there was a util­i­tar­ian TAI pro­ject along with a half-de­cent chance to do some­thing bet­ter (by my lights), I would ac­tively op­pose the util­i­tar­ian pro­ject. From my per­spec­tive, such a pro­ject is es­sen­tially en­emy com­bat­ants.

Just to be clear, this isn’t in re­sponse to some­thing I wrote, right? (I’m definitely not ad­vo­cat­ing any kind of “util­i­tar­ian TAI pro­ject” and would be quite scared of such a pro­ject my­self.)

More­over, from my per­spec­tive this kind of thing is hacks try­ing to work around the core is­sue, namely that I am not a util­i­tar­ian (along with the vast ma­jor­ity of peo­ple).

So what are you (and them) then? What would your utopia look like?

[Vanessa Kosoy]

Just to be clear, this isn’t in re­sponse to some­thing I wrote, right? (I’m definitely not ad­vo­cat­ing any kind of “util­i­tar­ian TAI pro­ject” and would be quite scared of such a pro­ject my­self.)

No! Sorry, if I gave that im­pres­sion.

So what are you (and them) then? What would your utopia look like?

Well, I linked my toy model of par­tial­ity be­fore. Are you ask­ing about some­thing more con­crete?

[Wei Dai]

Well, I linked my toy model of par­tial­ity be­fore. Are you ask­ing about some­thing more con­crete?

Yeah, I mean aside from how much you care about var­i­ous other peo­ple, what con­crete things do you want in your utopia?

[Vanessa Kosoy]

I have low con­fi­dence about this, but my best guess per­sonal utopia would be some­thing like: A lot of cool and in­ter­est­ing things are hap­pen­ing. Some of them are good, some of them are bad (a world in which noth­ing bad ever hap­pens would be bor­ing). How­ever, there is a limit on how bad some­thing is al­lowed to be (for ex­am­ple, true death, per­ma­nent crip­pling of some­one’s mind and eter­nal tor­ture are over the line), and over­all “happy end­ings” are more com­mon than “un­happy end­ings”. More­over, since it’s my utopia (ac­cord­ing to my un­der­stand­ing of the ques­tion, we are ig­nor­ing the bar­gain­ing pro­cess and acausal co­op­er­a­tion here), I am among the top along those de­sir­able di­men­sions which are zero-sum (e.g. play an es­pe­cially im­por­tant /​ “pro­tag­o­nist” role in the events to the ex­tent that it’s im­pos­si­ble for ev­ery­one to play such an im­por­tant role, and have high sta­tus to the ex­tent that it’s im­pos­si­ble for ev­ery­one to have such high sta­tus).

[Vanessa Kosoy]

First, you wrote “a part of me is ac­tu­ally more scared of many fu­tures in which al­ign­ment is solved, than a fu­ture where biolog­i­cal life is sim­ply wiped out by a pa­per­clip max­i­mizer.” So, I tried to as­suage this fear for a par­tic­u­lar class of al­ign­ment solu­tions.

Se­cond… Yes, for a util­i­tar­ian this doesn’t mean “much”. But, tbh, who cares? I am not a util­i­tar­ian. The vast ma­jor­ity of peo­ple are not util­i­tar­i­ans. Maybe even liter­ally no one is an (hon­est, not self-de­ceiv­ing) util­i­tar­ian. From my per­spec­tive, dis­ap­point­ing the imag­i­nary util­i­tar­ian is (in it­self) about as up­set­ting as dis­ap­point­ing the imag­i­nary pa­per­clip max­i­mizer.

Third, what I ac­tu­ally want from multi-user al­ign­ment is a solu­tion that (i) is ac­cept­able to me per­son­ally (ii) is ac­cept­able to the vast ma­jor­ity of peo­ple (at least if they think through it ra­tio­nally and are ar­gu­ing hon­estly and in good faith) (iii) is ac­cept­able to key stake­hold­ers (iv) as much as pos­si­ble, doesn’t leave any Pareto im­prove­ments on the table and (v) suffi­ciently Schel­ling-pointy to co­or­di­nate around. Here, “ac­cept­able” means “a lot bet­ter than pa­per­clips and not worth start­ing an AI race/​war to get some­thing bet­ter”.

[Wei Dai]

Se­cond… Yes, for a util­i­tar­ian this doesn’t mean “much”. But, tbh, who cares? I am not a util­i­tar­ian. The vast ma­jor­ity of peo­ple are not util­i­tar­i­ans. Maybe even liter­ally no one is an (hon­est, not self-de­ceiv­ing) util­i­tar­ian. From my per­spec­tive, dis­ap­point­ing the imag­i­nary util­i­tar­ian is (in it­self) about as up­set­ting as dis­ap­point­ing the imag­i­nary pa­per­clip max­i­mizer.

I’m not a util­i­tar­ian ei­ther, be­cause I don’t know what my val­ues are or should be. But I do as­sign sig­nifi­cant cre­dence to the pos­si­bil­ity that some­thing in the vinc­inity of util­i­tar­i­anism is the right val­ues (for me, or pe­riod). Given my un­cer­tain­ties, I want to ar­range the cur­rent state of the world so that (to the ex­tent pos­si­ble), what­ever I end up de­cid­ing my val­ues are, through things like rea­son, de­liber­a­tion, do­ing philos­o­phy, the world will ul­ti­mately not turn out to be a huge dis­ap­point­ment ac­cord­ing to those val­ues. Un­for­tu­nately, your pro­posed solu­tion isn’t very re­as­sur­ing to this kind of view.

It’s quite pos­si­ble that I (and peo­ple like me) are sim­ply out of luck, and there’s just no fea­si­ble way to do what we want to do, but it sounds like you think I shouldn’t even want what I want, or at least that you don’t want some­thing like this. Is it be­cause you’re already pretty sure what your val­ues are or should be, and there­fore think there’s lit­tle chance that mil­len­nia from now you’ll end up de­cid­ing that util­i­tar­i­anism (or NU, or what­ever) is right af­ter all, and re­gret not do­ing more in 2021 to push the world in the di­rec­tion of [your real val­ues, what­ever they are]?

[Vanessa Kosoy]

I’m mod­er­ately sure what my val­ues are, to some ap­prox­i­ma­tion. More im­por­tantly, I’m even more sure that, what­ever my val­ues are, they are not so ex­tremely differ­ent from the val­ues of most peo­ple that I should wage some kind of war against the ma­jor­ity in­stead of try­ing to ar­rive at a rea­son­able com­pro­mise. And, in the un­likely event that most peo­ple (in­clud­ing me) will turn out to be some kind of util­i­tar­i­ans af­ter all, it’s not a prob­lem: value ag­gre­ga­tion will then pro­duce a uni­verse which is pretty good for util­i­tar­i­ans.

[Wei Dai]

I’m mod­er­ately sure what my val­ues are, to some ap­prox­i­ma­tion. More im­por­tantly, I’m even more sure that, what­ever my val­ues are, they are not so ex­tremely differ­ent from the val­ues of most peo­ple [...]

Maybe you’re just not part of the tar­get au­di­ence of my OP then… but from my per­spec­tive, if I de­ter­mine my val­ues through the kind of pro­cess de­scribed in the first quote, and most peo­ple de­ter­mine their val­ues through the kind of pro­cess de­scribed in the sec­ond quote, it seems quite likely that the val­ues end up be­ing very differ­ent.

[...] that I should wage some kind of war against the ma­jor­ity in­stead of try­ing to ar­rive at a rea­son­able com­pro­mise.

The kind of solu­tion I have in mind is not “wag­ing war” but for ex­am­ple, solv­ing metaphilosos­phy and build­ing an AI that can en­courage philo­soph­i­cal re­flec­tion in hu­mans or en­hance peo­ple’s philo­soph­i­cal abil­ities.

And, in the un­likely pos­si­bil­ity that most peo­ple (in­clud­ing me) will turn out to be some kind of util­i­tar­i­ans af­ter all, it’s not a prob­lem: value ag­gre­ga­tion will then pro­duce a uni­verse which is pretty good for util­i­tar­i­ans.

What if you turn out to be some kind of util­i­tar­ian but most peo­ple don’t (be­cause you’re more like the first group in the OP and they’re more like the sec­ond group), or most peo­ple will even­tu­ally turn out to be some kind of util­i­tar­ian in a world with­out AI, but in a world with AI, this will hap­pen?

[Vanessa Kosoy]

I don’t think peo­ple de­ter­mine their val­ues through ei­ther pro­cess. I think that they already have val­ues, which are to a large ex­tent ge­netic and im­mutable. In­stead, these pro­cesses de­ter­mine what val­ues they pre­tend to have for game-the­ory rea­sons. So, the big differ­ence be­tween the groups is which “cards” they hold and/​or what strat­egy they pur­sue, not an in­trin­sic differ­ence in val­ues.

But also, if we do model val­ues as the re­sult of some long pro­cess of re­flec­tion, and you’re wor­ried about the AI dis­rupt­ing or in­suffi­ciently aid­ing this pro­cess, then this is already a sin­gle-user al­ign­ment is­sue and should be an­a­lyzed in that con­text first. The pre­sumed differ­ences in moral­ities are not the main source of the prob­lem here.

[Wei Dai]

I don’t think peo­ple de­ter­mine their val­ues through ei­ther pro­cess. I think that they already have val­ues, which are to a large ex­tent ge­netic and im­mutable. In­stead, these pro­cesses de­ter­mine what val­ues they pre­tend to have for game-the­ory rea­sons. So, the big differ­ence be­tween the groups is which “cards” they hold and/​or what strat­egy they pur­sue, not an in­trin­sic differ­ence in val­ues.

This is not a the­ory that’s fa­mil­iar to me. Why do you think this is true? Have you writ­ten more about it some­where or can link to a more com­plete ex­pla­na­tion?

But also, if we do model val­ues as the re­sult of some long pro­cess of re­flec­tion, and you’re wor­ried about the AI dis­rupt­ing or in­suffi­ciently aid­ing this pro­cess, then this is already a sin­gle-user al­ign­ment is­sue and should be an­a­lyzed in that con­text first. The pre­sumed differ­ences in moral­ities are not the main source of the prob­lem here.

This seems rea­son­able to me. (If this was meant to be an ar­gu­ment against some­thing I said, there may have been an­ther mis­com­munca­tion, but I’m not sure it’s worth track­ing that down.)

[Vanessa Kosoy]

This is not a the­ory that’s fa­mil­iar to me. Why do you think this is true? Have you writ­ten more about it some­where or can link to a more com­plete ex­pla­na­tion?

I con­sid­er­ing writ­ing about this for a while, but so far I don’t feel suffi­ciently mo­ti­vated. So, the links I posted up­wards in the thread are the best I have, plus vague ges­tur­ing in the di­rec­tions of Han­so­nian sig­nal­ing the­o­ries, Jaynes’ the­ory of con­scious­ness and Yud­kowsky’s be­lief in be­lief.

[mukashi]

Isn’t this the main the­sis of “The righ­teous mind”?

[Rafael Harth]

This com­ment seems to be con­sis­tent with the as­sump­tion that the out­come 1 year af­ter the sin­gu­lar­ity is locked in for­ever. But the fu­ture we’re dis­cussing here is one where hu­mans re­tain au­ton­omy (?), and in that case, they’re al­lowed to change their mind over time, es­pe­cially if hu­man­ity has ac­cess to a su­per­in­tel­li­gent al­igned AI. I think a fu­ture where we be­gin with highly sub­op­ti­mal per­sonal utopias and grad­u­ally tran­si­tion into util­itro­n­ium is among the more plau­si­ble out­comes. Com­pared with other out­comes where Not Every­one Dies, any­way. Your cre­dence may differ if you’re a moral rel­a­tivist.

[Wei Dai]

But the fu­ture we’re dis­cussing here is one where hu­mans re­tain au­ton­omy (?), and in that case, they’re al­lowed to change their mind over time, es­pe­cially if hu­man­ity has ac­cess to a su­per­in­tel­li­gent al­igned AI.

What if the hu­mans ask the al­igned AI to help them be more moral, and part of what they mean by “more moral” is hav­ing fewer doubts about their cur­rent moral be­liefs? This is what a “sta­tus game” view of moral­ity seems to pre­dict, for the hu­mans whose sta­tus games aren’t based on “do­ing philos­o­phy”, which seems to be most of them.

[Rafael Harth]

I don’t have any rea­son why this couldn’t hap­pen. My po­si­tion is some­thing like “moral­ity is real, prob­a­bly pre­cisely quan­tifi­able; seems plau­si­ble that in the sce­nario of hu­mans with au­ton­omy and al­igned AI, this could lead to an asym­me­try where more peo­ple tend to­ward util­itro­n­ium over time”. (Hence why I replied, you didn’t seem to con­sider that pos­si­bil­ity.) I could make up some mechanisms for this, but prob­a­bly you don’t need me for that. Also seems plau­si­ble that this doesn’t hap­pen. If it doesn’t hap­pen, maybe the peo­ple who get to de­cide what hap­pens with the rest of the uni­verse tend to­ward util­itro­n­ium. But my model is widely un­cer­tain and doesn’t rule out fu­tures of highly sub­op­ti­mal per­sonal utopias that per­sist in­definitely.

[Wei Dai]

I could make up some mechanisms for this, but prob­a­bly you don’t need me for that.

I’m in­ter­ested in your view on this, plus what we can po­ten­tially do to push the fu­ture in this di­rec­tion.

[Rafael Harth]

I strongly be­lieve that (1) well-be­ing is ob­jec­tive, (2) well-be­ing is quan­tifi­able, and (3) Open In­di­vi­d­u­al­ism is true (i.e., the con­cept of iden­tity isn’t well-defined, and you’re sub­jec­tively no less con­tin­u­ous with the fu­ture self if any other per­son than your own fu­ture self).

If (1-3) are all true, then util­itro­n­ium is the op­ti­mal out­come for ev­ery­one even if they’re en­tirely self­ish. Fur­ther­more, I ex­pect an AGI to figure this out, and to the ex­tent that it’s al­igned, it should com­mu­ni­cate that if it’s asked. (I don’t think an AGI will there­fore de­cide to do the right thing, so this is en­tirely com­pat­i­ble with ev­ery­one dy­ing if al­ign­ment isn’t solved.)

In the sce­nario where peo­ple get to talk to the AGI freely and it’s al­igned, two con­crete mechanisms I see are (a) peo­ple just ask the AGI what is morally cor­rect and it tells them, and (b) they get some small taste of what util­itro­n­ium would feel like, which would make it less scary. (A cru­cial piece is that they can ra­tio­nally ex­pect to ex­pe­rience this them­selves in the util­itro­n­ium fu­ture.)

In the sce­nario where peo­ple don’t get to talk to the AGI, who knows. It’s cer­tainly pos­si­ble that we have sin­gle­ton sce­nario with a few peo­ple in charge of the AGI, and they de­cide to cen­sor ques­tions about ethics be­cause they find the an­swers scary.

The only org I know of that works on this and shares my philo­soph­i­cal views is QRI. Their goal is to (a) come up with a math­e­mat­i­cal space (prob­a­bly a topolog­i­cal one, mb a Hilbert space) that pre­cisely de­scribes the sub­jec­tive ex­pe­rience of some­one, (b) find a way to put some­one in the scan­ner and cre­ate that space, and (c) find a prop­erty of that space that cor­re­sponds to their well-be­ing in that mo­ment. The flag ship the­ory is that this prop­erty is sym­me­try. Their model is stronger than (1-3), but if it’s cor­rect, you could get hard ev­i­dence on this be­fore AGI since it would make strong testable pre­dic­tions about peo­ple’s well-be­ing (and they think it could also point to easy in­ter­ven­tions, though I don’t un­der­stand how that works). Whether it’s fea­si­ble to do this be­fore AGI is a differ­ent ques­tion. I’d bet against it, but I think I give it bet­ter odds than any spe­cific al­ign­ment pro­posal. (And I hap­pen to know that Mike agrees that the fu­ture is dom­i­nated by con­cerns about AI and thinks this is the best thing to work on.)

So, I think their re­search is the best bet for get­ting more peo­ple on board with util­itro­n­ium since it can provide ev­i­dence on (1) and (2). (Also has the nice prop­erty that it won’t work if (1) or (2) are false, so there’s low risk of out­rage.) Other than that, write posts ar­gu­ing for moral re­al­ism and/​or for Open In­di­vi­d­u­al­ism.

Quan­tify­ing suffer­ing be­fore AGI would also plau­si­bly help with al­ign­ment, since at least you can for­mally spec­ify a broad space of out­comes you don’t want. though it cer­tainly doesn’t solve it, e.g. be­cause of in­ner op­ti­miz­ers.

[jacob_cannell]

This im­plies solv­ing a ver­sion of the al­ign­ment prob­lem that in­cludes rea­son­able value ag­gre­ga­tion be­tween differ­ent peo­ple (or be­tween AIs al­igned to differ­ent peo­ple),

We already have a solu­tion to this: money. It’s also the only solu­tion that satis­fies some es­sen­tial prop­er­ties such as sybil or­thog­o­nal­ity (es­pe­cially im­por­tant for posthu­man/​AGI so­cieties).

[TekhneMakre]

at least some re­searchers don’t seem to con­sider that part of “al­ign­ment”.

It’s part of al­ign­ment. Also, it seems mostly sep­a­rate from the part about “how do you even have con­se­quen­tial­ism pow­er­ful enough to make, say, nan­otech, with­out kil­ling ev­ery­one as a side-effect?”, and the lat­ter seems not too re­lated to the former.

[Duncan Sabien (Deactivated)]

In re­al­ity, ev­ery­one’s moral­ity is based on some­thing like the sta­tus game (see also: 1 2 3)

… I re­ally wanted to say [cita­tion needed], but then you did provide cita­tions, but then the cita­tions were not com­pel­ling to me.

I’m pretty op­posed to such uni­ver­sal claims be­ing made about hu­mans with­out push­back, be­cause such claims always seem to me to wish-away the ex­tremely wide vari­a­tion in hu­man psy­chol­ogy and the difficulty es­tab­lish­ing any­thing like “all hu­mans ex­pe­rience X.”

There are peo­ple who have no vi­sual imagery, peo­ple who do not think in words, peo­ple who have no sense of con­ti­nu­ity of self, peo­ple who have no dis­cernible emo­tional re­sponse to all sorts of “emo­tional” stim­uli, and on and on and on.

So, I’ll go with “it makes sense to model peo­ple as if ev­ery one of them is mo­ti­vated by struc­tures built atop the sta­tus game.” And I’ll go with “it seems like the sta­tus ar­chi­tec­ture is a phys­iolog­i­cal near-uni­ver­sal, so I have a hard time imag­in­ing what else peo­ple’s moral­ity might be made of.” And I’ll go with “ev­ery­one I’ve ever talked to had moral­ity that seemed to me to cash out to be­ing sta­tusy, ex­cept the peo­ple whose self-re­ports I ig­nored be­cause they didn’t fit the story I was build­ing in my head.”

But I re­ject the blunt uni­ver­sal for not even pre­tend­ing that it’s in­ter­ested in mak­ing it­self falsifi­able.

[Wei Dai]

Kind of frus­trat­ing that this high karma re­ply to a high karma com­ment on my post is based on a dou­ble mi­s­un­der­stand­ing/​mis­com­mu­ni­ca­tion:

1. First Vanessa un­der­stood me as claiming that a sig­nifi­cant num­ber of peo­ple’s moral­ity is not based on sta­tus games. I tried to clar­ify in an ear­lier com­ment already, but to clar­ify some more: that’s not my in­tended dis­tinc­tion be­tween the two groups. Rather the dis­tinc­tion is that the first group “know or at least sus­pect that they are con­fused about moral­ity, and are ea­ger or will­ing to ap­ply rea­son and de­liber­a­tion to find out what their real val­ues are, or to cor­rect their moral be­liefs” (they can well be do­ing this be­cause of the sta­tus game that they’re play­ing) whereas this quoted de­scrip­tion doesn’t ap­ply to the sec­ond group.

2. Then you (Dun­can) un­der­stood Vanessa as claiming that liter­ally ev­ery­one’s moral­ity is based on sta­tus games, when (as the sub­se­quent dis­cus­sion re­vealed) the in­tended mean­ing was more like “the num­ber of peo­ple whose moral­ity is not based on sta­tus games is a lot fewer than (Vanessa’s mi­s­un­der­stand­ing of) Wei’s claim”.

[Duncan Sabien (Deactivated)]

I think it’s im­por­tant and valuable to sep­a­rate out “what was in fact in­tended” (and I straight­for­wardly be­lieve Vanessa’s restate­ment as a truer ex­pla­na­tion of her ac­tual po­si­tion) from “what was origi­nally said, and how would 70+ out of 100 read­ers tend to in­ter­pret it.”

I think we’ve cleared up what was meant. I still think it was bad that [the perfectly rea­son­able thing that was meant] was said in a [pre­dictably mis­lead­ing fash­ion].

But I think we’ve said all that needs to be said about that, too.

[Said Achmiz]

This is a tan­gent (so maybe you pre­fer to di­rect this dis­cus­sion el­se­where), but: what’s with the brack­ets? I see you us­ing them reg­u­larly; what do they sig­nify?

[Duncan Sabien (Deactivated)]

I use them where I’m try­ing to con­vey a sin­gle noun that’s made up of many words, and I’m scared that peo­ple will lose track of the over­all sen­tence while in the mid­dle of the chunk. It’s an at­tempt to keep the over­all sen­tence un­der­stand­able. I’ve tried hy­phen­at­ing such phrases and peo­ple find that more an­noy­ing.

[Said Achmiz]

Hmm, I see, thanks.

[Vanessa Kosoy]

It’s not just that the self-re­ports didn’t fit the story I was build­ing, the self-re­ports didn’t fit the re­vealed prefer­ences. What­ever peo­ple say about their moral­ity, I haven’t seen any­one who be­haves like a true util­i­tar­ian.

IMO, this is the source of all the gnash­ing of teeth about how much % of your salary you need to donate: the fun­da­men­tal con­tra­dic­tion be­tween the de­mands of util­i­tar­i­anism and how much peo­ple are ac­tu­ally will­ing to pay for the sta­tus gain. Ofc many ex­cuses were de­vel­oped (“sure I still need to buy that coffee or those movie tick­ets, oth­er­wise I won’t be pro­duc­tive”) but they don’t sound like the most par­si­mo­nious ex­pla­na­tion.

This is also the source of para­doxes in pop­u­la­tion ethics and its vicinity: those ab­strac­tions are just very re­mote from ac­tual hu­man minds, so there’s no rea­son they should pro­duce any­thing sane in edge cases. Their only true util­ity is as an ap­prox­i­mate guideline for mak­ing group de­ci­sions, for suffi­ciently mun­dane sce­nar­ios. Once you get to is­sues with in­fini­ties it be­comes clear util­i­tar­i­anism is not even math­e­mat­i­cally co­her­ent, in gen­eral.

You’re right that there is a lot of vari­a­tion in hu­man psy­chol­ogy. But it’s also an ac­cepted prac­tice to phrase claims as uni­ver­sal when what you ac­tu­ally mean is, the ex­cep­tions are neg­ligible for our prac­ti­cal pur­pose. For ex­am­ple, most peo­ple would ac­cept “hu­mans have 2 arms and 2 legs” as a true state­ment in many con­texts, even though some hu­mans have less. In this case, my claim is that the ex­cep­tions are much rarer than the OP seems to im­ply (i.e. most peo­ple the OP clas­sifies as ex­cep­tions are not re­ally ex­cep­tions).

I’m all for falsifi­a­bil­ity, but it’s gen­uinely hard to do falsifi­a­bil­ity in soft top­ics like this, where no the­ory makes very sharp pre­dic­tions and col­lect­ing data is hard. Ul­ti­mately, which ex­pla­na­tion is more rea­son­able is go­ing to be at least in part an in­tu­itive judge­ment call based on your own ex­pe­rience and re­flec­tion. So, yes, I cer­tainly might be wrong, but what I’m de­scribing is my cur­rent best guess.

[Duncan Sabien (Deactivated)]

But it’s also an ac­cepted prac­tice to phrase claims as uni­ver­sal when what you ac­tu­ally mean is, the ex­cep­tions are neg­ligible for our prac­ti­cal pur­pose. For ex­am­ple, most peo­ple would ac­cept “hu­mans have 2 arms and 2 legs” as a true state­ment in many con­texts, even though some hu­mans have less.

The equiv­a­lent state­ment would be “In re­al­ity, ev­ery­one has 2 arms and 2 legs.”

[Vanessa Kosoy]

Well, if the OP said some­thing like “most peo­ple have 2 eyes but en­light­ened Bud­dhists have a third eye” and I re­sponded with “in re­al­ity, ev­ery­one have 2 eyes” then, I think my mean­ing would be clear even though it’s true that some peo­ple have 1 or 0 eyes (afaik maybe there is even a rare mu­ta­tion that cre­ates a real third eye). Not adding all pos­si­ble qual­ifiers is not the same as “not even pre­tend­ing that it’s in­ter­ested in mak­ing it­self falsifi­able”.

[Duncan Sabien (Deactivated)]

I think your mean­ing would be clear, but “ev­ery­one knows what this straight­for­wardly false thing that I said re­ally meant” is in­suffi­cient for a sub­cul­ture try­ing to be pre­cise and ac­cu­rate and con­verge on truth. Seems like more LWers are on your side than on mine on that ques­tion, but that’s not news. ¯\_(ツ)_/​¯

It’s a straw­man to pre­tend that “please don’t say a clearly false thing” is me in­sist­ing on “please in­clude all pos­si­ble qual­ifiers.” I just wish you hadn’t said a clearly false thing, is all.

[Vanessa Kosoy]

Nat­u­ral lan­guage is not math, it’s in­her­ently am­bigu­ous and it’s not re­al­is­ti­cally pos­si­ble to always be pre­cise with­out im­plic­itly as­sum­ing any­thing about the reader’s un­der­stand­ing of the con­text. That said, it seems like I wasn’t suffi­ciently pre­cise in this case, so I ed­ited my com­ment. Thank you for the cor­rec­tion.

[Vladimir_Nesov]

in­suffi­cient for a sub­cul­ture try­ing to be pre­cise and ac­cu­rate and con­verge on truth

The trade­off is with ver­bosity and difficulty of com­mu­ni­ca­tion, it’s not always a straight­for­ward Pareto im­prove­ment. So in this case I fully agree with drop­ping “ev­ery­one” or re­plac­ing it with a more ac­cu­rate qual­ifier. But I dis­agree with a gen­eral prin­ci­ple that would dis­count ease for a per­son who is trained and tal­ented in rele­vant ways. New habits of thought that be­come in­tu­itive are im­prove­ments, check­lists and other de­liber­a­tive rit­u­als that slow down think­ing need merit that over­comes their con­sid­er­able cost.

[Gunnar_Zarncke]

I haven’t seen any­one who be­haves like a true util­i­tar­ian.

That looks like a No True Scots­man ar­gu­ment to me. Just be­cause the ex­treme doesn’t ex­ist doesn’t mean that all of the scale can be ex­plained by sta­tus games.

[Vanessa Kosoy]

What does it have to do with “No True Scots­man”? NTS is when you re­define your cat­e­gories to jus­tify your claim. I don’t think I did that any­where.

Just be­cause the ex­treme doesn’t ex­ist doesn’t mean that all of the scale can be ex­plained by sta­tus games.

First, I didn’t say all the scale is ex­plained by sta­tus games, I did men­tion em­pa­thy as well.

Se­cond, that by it­self sure doesn’t mean much. Ex­plain­ing all the ev­i­dence would re­quire an ar­ti­cle, or maybe a book (al­though I hoped the posts I linked ex­plain some of it). My point here is that there is an enor­mous dis­crep­ancy be­tween the re­ported moral­ity and the re­vealed prefer­ences, so be­liev­ing self-re­ports is clearly a non-starter. How do you build an ex­pla­na­tion not from self-re­ports is a differ­ent (long) story.

[Gunnar_Zarncke]

I agree that there is an enor­mous dis­crep­ancy.

[Ratios]

If you try to quan­tify it, hu­mans on av­er­age prob­a­bly spend over 95% (Con­ser­va­tive es­ti­ma­tion) of their time and re­sources on non-util­i­tar­ian causes. True util­i­tar­ian be­hav­ior Is ex­tremely rare and all other moral be­hav­iors seem to be ei­ther elab­o­rate sta­tus games or ex­tended self-in­ter­est [1]. The typ­i­cal hu­man is way closer un­der any rele­vant quan­tified KPI to be­ing com­pletely self­ish than be­ing a util­i­tar­ian.

[1] - In­vest­ing in your fam­ily/​friends is in a way self­ish, from a genes/​al­li­ances (re­spec­tively) per­spec­tive.

[Sweetgum]

But, I still pre­fer that over pa­per­clips (by far). And, I sus­pect that most peo­ple do (even if they protest it in or­der to play the game).

What does this even mean? If some­one says they don’t want X, and they never take ac­tions that pro­mote X, how can it be said that they “truly” want X? It’s not their stated prefer­ence or their re­vealed prefer­ence!

[Viliam]

Fem­i­nist hero and birth con­trol cam­paigner Marie Stopes, who was voted Wo­man of the Millen­nium by the read­ers of The Guardian and hon­oured on spe­cial Royal Mail stamps in 2008, was an anti-Semite and eugenicist

My con­clu­sion from this is more like “suc­cess­ful poli­ti­ci­ans are not moral paragons”. More gen­er­ally, try­ing to find morally vir­tu­ous peo­ple by a pop­u­lar vote is not go­ing to pro­duce great re­sults, be­cause the pop­u­lar­ity plays much greater role than moral­ity.

I googled for “woman of the year” to get more data points; found this list, con­tain­ing: 2019 Greta Thun­berg, 2016 Hillary Clin­ton, 2015 An­gela Merkel, 2010 Nancy Pelosi, 2008 Michelle Obama, 1999 Madeleine Albright, 1990 Aung San Suu Kyi… clearly, be­ing a poli­ti­cian dra­mat­i­cally in­creases your chances of win­ning. Look­ing at their be­hav­ior, Aung San Suu Kyi later or­ga­nized a geno­cide.

The list also in­cludes 2009 Malala Yousafzai, who as far as I know is an ac­tual hero with no dark side. But that’s kinda my point, that putting Malala Yousafzai on the same list as Greta Thun­berg and Hillary Clin­ton just makes the list con­fus­ing. And if you had to choose one of them as the “woman of the mil­le­nium”, I would ex­pect most read­ers to vote for some­one rep­re­sent­ing their poli­ti­cal tribe. But to me that does not mean that peo­ple have no sense of moral­ity, only that they can eas­ily get poli­ti­cally mind­kil­led.

For the Mala­gasy peo­ple in Mada­gas­car, it’s taboo [...] to sleep fac­ing west­wards, as you’ll kick the sun­rise.

And this sounds silly to us, be­cause we know that “kick­ing the sun­rise” is im­pos­si­ble, be­cause Sun is a planet, it is far away, and your kick­ing has no im­pact on it.

So, we should dis­t­in­guish be­tween peo­ple hav­ing differ­ent moral feel­ings, and hav­ing differ­ent mod­els of the world. If you ac­tu­ally be­lieved that kick­ing the Sun is pos­si­ble and can have as­tro­nom­i­cal con­se­quences, you would prob­a­bly also per­ceive peo­ple sleep­ing west­wards as crim­i­nally neg­li­gent, pos­si­bly psy­cho­pathic.

Kinda like be­ing an­gry at peo­ple who don’t wear face masks only makes sense un­der the as­sump­tion that the face masks pre­vent spread­ing of a po­ten­tially deadly dis­ease. Without this con­text, anger to­wards peo­ple with no face masks would just be silly.

[Wei Dai]

And this sounds silly to us, be­cause we know that “kick­ing the sun­rise” is im­pos­si­ble, be­cause Sun is a planet, it is far away, and your kick­ing has no im­pact on it.

I think a lot of con­tem­po­rary cul­tures back then would have found “kick­ing the sun­rise” to be silly, be­cause it was ob­vi­ously im­pos­si­ble even given what they knew at the time, i.e., you can only kick some­thing if you phys­i­cally touch it with your foot, and no­body has ever even got­ten close to touch­ing the sun, and it’s even more im­pos­si­ble while you’re asleep.

So, we should dis­t­in­guish be­tween peo­ple hav­ing differ­ent moral feel­ings, and hav­ing differ­ent mod­els of the world. If you ac­tu­ally be­lieved that kick­ing the Sun is pos­si­ble and can have as­tro­nom­i­cal con­se­quences, you would prob­a­bly also per­ceive peo­ple sleep­ing west­wards as crim­i­nally neg­li­gent, pos­si­bly psy­cho­pathic.

Why did the Mala­gasy peo­ple have such a silly be­lief? Why do many peo­ple have very silly be­liefs to­day? (Among the least poli­ti­cally risky ones to cite, some­one I’ve known for years who oth­er­wise is in­tel­li­gent and suc­cess­ful, cur­rently be­lieves, or at least be­lieved in the re­cent past, that ²⁄₃ of ev­ery­one will die as a re­sult of tak­ing the COVID vac­cines.) I think the un­for­tu­nate an­swer is that peo­ple are mo­ti­vated to or are re­li­ably caused to have cer­tain false be­liefs, as part of the sta­tus games that they’re play­ing. I wrote about one such dy­namic, but that’s prob­a­bly not a com­plete ac­count.

[null]

morally vir­tu­ous people

I feel like your defi­ni­tion of “morally vir­tu­ous” is miss­ing at least 2 pa­ram­e­ters: the con­text that the per­son is in, and the defi­ni­tion of “morally vir­tu­ous”. You seem to treat both as fixed or not con­tribut­ing to the out­come, but in my ex­pe­rience they’re equally if not more im­por­tant than the per­son. Your ex­am­ple of Aung San Suu Kyi is a good ex­am­ple of that. She was “good” in 1990 given her in­cen­tives in 1990 and the pop­u­lar defi­ni­tion of “good” in 1990. Not so much later.

[Viliam]

Mo­ral virtue seems to in­volve cer­tain… in­flex­i­bil­ity to in­cen­tives.

If some­one says “I would or­ga­nize the geno­cide of Ro­hingya if and only if or­ga­niz­ing such geno­cide is prof­itable, and it so hap­pens that to­day it would be un­prof­itable, there­fore to­day I op­pose the geno­cide”, we would typ­i­cally not call this per­son moral.

Of course, peo­ple usu­ally do not ex­plain their de­ci­sion al­gorithms in de­tail, so the per­son de­scribed above would prob­a­bly only say “I op­pose the geno­cide”, which would seem quite nice of them.

With most peo­ple, we will never know what they would do in a par­allel uni­verse, where or­ga­niz­ing a geno­cide could give them a well-paid job. Without ev­i­dence to con­trary, we usu­ally char­i­ta­bly as­sume that they would re­fuse… but of course, per­haps this is un­re­al­is­ti­cally op­ti­mistic.

(This only ad­dresses the ob­jec­tion about “con­text”. The prob­lem of defi­ni­tion is more com­pli­cated.)

[fourier]

> and this sounds silly to us, be­cause we know that “kick­ing the sun­rise” is im­pos­si­ble, be­cause sun is a planet, it is far away, and your kick­ing has no im­pact on it.

No, the rea­son it sounds silly to you is not be­cause it’s not true, but be­cause it’s not part of your own sa­cred be­liefs. There is no fun­da­men­tal rea­son for peo­ple to sup­port things you are tak­ing for granted as moral facts, like women’s right or racial rights.

In fact, given an ac­cu­rate model of the world, a lot of things that make the most sense you may find dis­taste­ful based on your cur­rent un­usual “moral” fash­ions.

For ex­am­ple, ex­ter­mi­nat­ing op­pos­ing groups is com­mon in hu­man so­cieties his­tor­i­cally. Often groups are com­pet­ing for re­sources, since one group wants more re­sources for them and their progeny, ex­ter­mi­nat­ing the other group makes the most sense.

And if the fun­da­men­tal de­sire for sur­vival and dom­i­nance—drilled into us by evolu­tion—isn’t moral, then the con­cept just seems to­tally mean­ingless.

[Viliam]

And if the fun­da­men­tal de­sire for sur­vival and dom­i­nance—drilled into us by evolu­tion—isn’t moral, then the con­cept just seems to­tally mean­ingless.

A con­cept is “to­tally mean­ingless” just be­cause it does not match some evolu­tion­ary strate­gies? First, con­cepts are con­cepts, re­gard­less of what is their re­la­tion to evolu­tion. Se­cond, there are many strate­gies in evolu­tion, in­clud­ing things like co­op­er­a­tion or com­mit­ments, which in­tu­itively seem more al­igned with moral­ity.

Hu­mans are a so­cial species, where the most ag­gre­sive one with most mus­cles is not nec­es­sar­ily a win­ner. Some­times it is ac­tu­ally a loser, who gets beaten by the cops and thrown in jail. Another ex­am­ple: Some home­less peo­ple are quite scary and they can sur­vive things that I prob­a­bly can­not imag­ine; yet, from the evolu­tion­ary per­spec­tive, they are usu­ally less suc­cess­ful than me.

Even if a group wants to ex­ter­mi­nate an­other group, it is usu­ally eas­ier if they befriend a differ­ent group first, and then at­tack to­gether. But you usu­ally don’t make friends by be­ing a back­stab­bing ass­hole. And “not be­ing a back­stab­bing ass­hole” is kinda what moral­ity is about.

There is no fun­da­men­tal rea­son for peo­ple to sup­port things you are tak­ing for granted as moral facts, like women’s right or racial rights.

Here we need to de­cou­ple moral prin­ci­ples from fac­tual be­liefs. On the level of moral prin­ci­ples, many peo­ple ac­cept “if some in­di­vi­d­ual is similar to me, they should be treated with some ba­sic re­spect” as a moral rule. Not all of them, of course. If some­one does not ac­cept this moral rule, then… de gustibus non est dis­putan­dum, I guess. (I sus­pect that ethics is some­how down­stream of aes­thet­ics, but I may be con­fused about this.) But even if some­one ac­cepts this rule, the ac­tual ap­pli­ca­tion will de­pend on their fac­tual be­liefs about who is “similar to me”.

I be­lieve it is a state­ment about the world (not just some kind of sa­cred be­lief) that ap­proval of women’s rights is pos­i­tively cor­re­lated with the be­lief that (men­tally) women are similar to men. Similarly, the ap­proval of racial rights is pos­i­tively cor­re­lated with the be­lief that peo­ple of differ­ent races are (men­tally) similar to each other. This state­ment should be some­thing that both peo­ple who ap­prove and who dis­ap­prove of the afore­men­tioned rights should agree upon.

At least it seems to me that his­tor­i­cally, peo­ple who pro­moted these rights of­ten ar­gued about similar­ity; and peo­ple who op­posed these rights of­ten ar­gued about dis­similar­ity. For ex­am­ple, if you be­lieve that women are in­her­ently in­ca­pable of ab­stract think­ing, then of course it does not make any sense to let them study at uni­ver­si­ties. Or if you be­lieve that black peo­ple en­joy be­ing slaves, and ac­tu­ally slav­ery is much bet­ter for them than free­dom, then of course abo­li­tion­ists are just evil fa­nat­ics. But if it turns of that these be­liefs are fac­tu­ally wrong, then this be­lief up­date has moral con­se­quences. If does not effect which moral prin­ci­ples you ac­cept; but if you already ac­cept some moral prin­ci­ples (and many peo­ple do) it can effect what these moral prin­ci­ples ap­ply to. You can be­come an X rights pro­po­nent not by adopt­ing a new moral prin­ci­ple, but by learn­ing that your already ex­ist­ing moral prin­ci­ple ac­tu­ally also ap­plies to group X (and then it re­quires some moral pres­sure to over­come com­part­men­tal­iza­tion).

This agains if differ­ent from the ques­tion what is the right mean­ing of the word “similar” in sen­tence “peo­ple similar to me should be treated with re­spect”. What kinds of similar­ity mat­ter? Is the color of the eyes im­por­tant? Or is it more about be­ing sen­tient, ca­pa­ble of feel­ing pain, and such stuff? Again, it seems to me that if some­one de­cides that the color of the eyes is ul­ti­mately unim­por­tant, that per­son is not mak­ing a com­pletely ran­dom de­ci­sion, but some­one builds on the already ex­ist­ing un­der­ly­ing moral feel­ings (per­haps com­bin­ing them with some fac­tual be­liefs about how the eye color is or isn’t re­lated to other things that mat­ter).

[Jon Garcia]

Even if moral­ities vary from cul­ture to cul­ture based on the lo­cal sta­tus games, I would sug­gest that there is still some amount of con­se­quen­tial­ist bedrock to why cer­tain types of norms de­velop. In other words, cul­tural rel­a­tivism is not un­bounded.

Gen­er­ally speak­ing, norms evolve over time, where any given norm at one point didn’t yet ex­ist if you go back far enough. What caused these norms to de­velop? I would say the se­lec­tive pres­sures for norm de­vel­op­ment come from some com­bi­na­tion of ex­ist­ing cul­turally-spe­cific norms and nar­ra­tives (such as the sun­rise be­ing an agent that could get hurt when kicked) along with more hu­man-uni­ver­sal mo­ti­va­tions (such as em­pa­thy + {wellbe­ing = good, suffer­ing = bad} → you are bad for kick­ing the sun­rise → don’t sleep fac­ing west) or other in­stru­men­tally-con­ver­gent goals (such as {power = good} + “se­men grants power” → in­sti­tu­tion­al­ized sodomy). At ev­ery step along the evolu­tion of a moral norm, ev­ery change needs to be jus­tifi­able (in a con­se­quen­tial­ist sense) to the mem­bers of the com­mu­nity who would adopt it. Mo­ral progress is when the norms of so­ciety come to bet­ter res­onate with both the ac­cepted nar­ra­tives of so­ciety (which may come from leg­ends or from sci­ence) and the in­trin­sic val­ues of its mem­bers (which come from our biol­ogy /​ psy­chol­ogy).

In a world where al­ign­ment has been solved to most ev­ery­one’s satis­fac­tion, I think that the sta­tus-game /​ cul­tural nar­ra­tive as­pect of moral­ity will nec­es­sar­ily have been taken into ac­count. For ex­am­ple, imag­ine a post-Sin­gu­lar­ity world kind of like Scott Alexan­der’s Archipelago, where the ASI co­op­er­ates with each sub-com­mu­nity to cre­ate a cus­tomized nar­ra­tive for the mem­bers to par­ti­ci­pate in. It might then slowly ad­just this nar­ra­tive (over decades? cen­turies?) to al­ign bet­ter with hu­man flour­ish­ing in other di­men­sions. The sta­tus-game as­pect could re­main in play as long as sta­tus be­comes suffi­ciently cor­re­lated with some­thing like “uses their role in life to im­prove the lives of oth­ers within their sphere of con­trol”. And I think ev­ery­one would be bet­ter off if each nar­ra­tive also be­comes at least con­sis­tent with what we learn from sci­ence, even though the sto­ries that define the sta­tus game will be differ­ent from one cul­ture to an­other in other ways.

[Wei Dai]

Upvoted for some in­ter­est­ing thoughts.

In a world where al­ign­ment has been solved to most ev­ery­one’s satis­fac­tion, I think that the sta­tus-game /​ cul­tural nar­ra­tive as­pect of moral­ity will nec­es­sar­ily have been taken into ac­count. For ex­am­ple, imag­ine a post-Sin­gu­lar­ity world kind of like Scott Alexan­der’s Archipelago, where the ASI co­op­er­ates with each sub-com­mu­nity to cre­ate a cus­tomized nar­ra­tive for the mem­bers to par­ti­ci­pate in. It might then slowly ad­just this nar­ra­tive (over decades? cen­turies?) to al­ign bet­ter with hu­man flour­ish­ing in other di­men­sions.

Can you say more about how you see us get­ting from here to there?

[Jon Garcia]

Get­ting from here to there is always the tricky part with co­or­di­na­tion prob­lems, isn’t it? I do have some (quite spec­u­la­tive) ideas on that, but I don’t see hu­man so­ciety or­ga­niz­ing it­self in this way on its own for at least a few cen­turies given cur­rent poli­ti­cal and eco­nomic trends, which is why I pos­tu­lated a co­op­er­a­tive ASI.

So as­sum­ing that ei­ther an al­igned ASI has taken over (I have some ideas on ro­bust al­ign­ment, too, but that’s out of scope here) or poli­ti­cal and eco­nomic forces (and in­fras­truc­ture) have fi­nally pushed hu­man­ity past a cer­tain so­cial phase tran­si­tion, I see hu­man­ity un­der­go­ing an or­ga­ni­za­tional shift much like what hap­pened with the evolu­tion of mul­ti­cel­lu­lar­ity and eu­so­cial­ity. This would look at first mostly the same as to­day, ex­cept that na­tional bor­ders have be­come mostly ir­rele­vant due to ad­vances in trans­porta­tion and com­mu­ni­ca­tion in­fras­truc­ture. Ba­si­cally, imag­ine the world’s cities and high­ways be­com­ing some­thing like the vas­cu­lar sys­tem of di­cots or the closed cir­cu­la­tory sys­tem of ver­te­brates, with the re­gions en­closed by net­work cir­cuits act­ing as de facto states (or or­gans/​tis­sues, to con­tinue the biolog­i­cal anal­ogy). Ma­jor cities and the re­gions along the high­ways that con­nect them be­come the de facto ar­biters of in­ter­na­tional policy, while the ma­jor cities and high­ways within each re­gion be­come the ar­biters of re­gional policy, and so on in a hi­er­ar­chi­cally em­bed­ded man­ner.

Within this struc­ture, en­closed re­gions would act as hi­er­ar­chi­cally em­bed­ded com­mu­ni­ties that end up perform­ing a di­vi­sion of la­bor for the global net­work, just as or­gans di­vide la­bor for the body (or like tis­sues di­vide la­bor within an or­gan, or cells within a tis­sue, or or­ganel­les within a cell, if you’re look­ing within re­gions). Ba­si­cally, the trans­porta­tion/​com­mu­ni­ca­tion/​etc. net­work edges would come to act as Markov blan­kets for the re­gions they en­cap­su­late, and this or­ga­ni­za­tion would ex­tend hi­er­ar­chi­cally, just like in biolog­i­cal sys­tems, down to the level of lo­cal com­mu­ni­ties. (Ideally, each com­mu­nity would be­come lo­cally self-suffi­cient, with broader net­works tak­ing on a more mod­u­la­tory role, but that’s an­other dis­cus­sion.)

Any­way, once this point is reached, or even as the tran­si­tion is un­der­way, I see the ASI and/​or so­cial pres­sures fa­cil­i­tat­ing the move­ment of peo­ple to­ward com­mu­ni­ties of shared val­ues and be­liefs (i.e., shared nar­ra­tives, or at least min­i­mally con­flict­ing nar­ra­tives), much like in Scott Alexan­der’s Archipelago. Each per­son or fam­ily unit should move so as to min­i­mize their dis­place­ment while max­i­miz­ing the marginal im­prove­ment they could make to their new com­mu­nity (and the marginal benefit they could re­ceive from the new com­mu­nity).

In the sys­tem that emerges, sto­ries would be­come some­thing of a com­mod­ity, aris­ing within com­mu­ni­ties as shared nar­ra­tives that as­sign so­cial roles and teach val­ues and les­sons (just like the campfire leg­ends of an­cient hunter-gath­erer so­cieties). Sto­ries with more uni­ver­sal res­o­nance would prop­a­gate up hi­er­ar­chi­cal lay­ers of the global net­work and then get dis­sem­i­nated top-down to­ward other lo­cal com­mu­ni­ties within the broader re­gions. This would provide a nar­ra­tive-syn­chro­niza­tion effect at high lev­els and across ad­ja­cent re­gions while also al­low­ing for lo­cal vari­a­tions. The sta­tus games /​ moral­ities of the in­ter­na­tional level would even­tu­ally at­tain a more “liberal” fla­vor, while those at more lo­cal lev­els could be more “con­ser­va­tive” in na­ture.

Sorry, that was long. And it prob­a­bly in­volved more ideal­iza­tional fan­tasy than ra­tio­nal pre­dic­tion of fu­ture trends. But I have a hunch that some­thing like this could work

[moridinamael]

I am also scared of fu­tures where “al­ign­ment is solved” un­der the cur­rent pre­vailing us­age of “hu­man val­ues.”

Hu­mans want things that we won’t end up lik­ing, and pre­fer things that we will re­gret get­ting rel­a­tive to other op­tions that we pre­vi­ously dis­preferred. We are re­mark­ably ig­no­rant of what we will, in ret­ro­spect, end up hav­ing liked, even over short timescales. Over longer timescales, we learn to like new things that we couldn’t have pre­dicted a pri­ori, mean­ing that even our earnest and thought­fully-con­sid­ered best guess of our prefer­ences in ad­vance will pre­dictably be a mis­match for what we would have preferred in ret­ro­spect.

And this is not some kind of bug, this is cen­trally im­por­tant to what it is to be a per­son; “grow­ing up” re­quires a con­stant pro­cess of learn­ing that you don’t ac­tu­ally like cer­tain things you used to like and now sud­denly like new things. This truth ranges over all are­nas of ex­is­tence, from learn­ing to like black coffee to re­al­iz­ing you want to have chil­dren.

I am per­son­ally par­tial to the idea of some­thing like Co­her­ent Ex­trap­o­lated Vo­li­tion. But it seems sus­pi­cious that I’ve never seen any­body on LW sketch out how a de­ci­sion the­ory ought to be­have in situ­a­tions where the agents util­ity func­tion will have pre­dictably changed by the time the out­come ar­rives so the “best choice” is ac­tu­ally a cur­rently dis­preferred choice. (In other words, situ­a­tions where the “best choice” in ret­ro­spect, and in ex­pec­ta­tion, do not match.) It seems dan­ger­ous to throw our­selves into a fu­ture where “best-in-ret­ro­spect” wins ev­ery time, be­cause I can imag­ine many al­ter­a­tions to my util­ity func­tion that I definitely wouldn’t want to ac­cept in ad­vance, but which would make me “hap­pier” in the end. And it also seems awful to ac­cept a pro­cess by which “best-in-ex­pec­ta­tion” wins ev­ery time, be­cause I think a likely re­sult is that we are frozen into what­ever our cur­rent util­ity func­tion looks like for­ever. And I do not see any prin­ci­pled and philo­soph­i­cally ob­vi­ous method by which we ought to ar­bi­trate be­tween in-ad­vance and in-ret­ro­spect prefer­ences.

Another way of say­ing the above is that it seems that “want­ing” and “lik­ing” ought to co­here but how they ought to co­here seems tricky to define with­out bak­ing in some ques­tion-beg­ging as­sump­tions.

[Astor]

I thought a solved al­ign­ment prob­lem would im­pli­cate a con­stant pro­cess of chang­ing the val­ues of the AI in re­gard to the most re­cent hu­man val­ues. So if some­thing does not lead to the ex­pected ter­mi­nal goals of the hu­man (such as en­joy­able emo­tions), then the hu­man can in­di­cate that out­come to the AI and the AI would ad­just its own goals ac­cord­ingly.

[moridinamael]

The idea that the AI should defer to the “most re­cent” hu­man val­ues is an in­stance of the sort of trap I’m wor­ried about. I sus­pect we could be led down an in­cre­men­tal path of small value changes in prac­ti­cally any di­rec­tion, which could ter­mi­nate in our will­ing and ea­ger self-ex­tinc­tion or per­ma­nent wire­head­ing. But how much tyranny should pre­sent-hu­man­ity be al­lowed to have over the choices of fu­ture hu­man­ity?

I don’t think “none” is as wise an an­swer as it might sound at first. To an­swer “none” im­plies a kind of moral rel­a­tivism that none of us ac­tu­ally hold, and which would make us merely the au­thors of a pro­cess that ul­ti­mately de­stroys ev­ery­thing we cur­rently value.

But also, the an­swer of “com­plete con­trol by the fu­ture by the pre­sent” seems ob­vi­ously wrong, be­cause we will learn about en­tirely new things worth car­ing about that we can’t pre­dict now, and some­times it is nat­u­ral to change what we like.

More fun­da­men­tally, I think the as­sump­tion that there ex­ist “hu­man ter­mi­nal goals” pre­sumes too much. Speci­fi­cally, it’s an as­sump­tion that pre­sumes that our de­sires, in an­ti­ci­pa­tion and in ret­ro­spect, are des­tined to fun­da­men­tally and pre­dictably co­here. I would bet money that this isn’t the case.

[Vladimir_Nesov]

The im­pli­ca­tion of do­ing ev­ery­thing that AI could do at once is un­for­tu­nate. The ur­gent ob­jec­tive of AI al­ign­ment is pre­ven­tion of AI risk, where a min­i­mal solu­tion is to take away ac­cess to un­re­stricted com­pute from all hu­mans in a cor­rigible way that would al­low even­tual de­sir­able use of it. All other ap­pli­ca­tions of AI could fol­low much later through cor­rigi­bil­ity of this ur­gent ap­pli­ca­tion.

[deleted]

[deleted]

[moridinamael]

Yes, there is a broad class of wire­head­ing solu­tions that we would want to avoid, and it is not clear how to spec­ify a rule that dis­t­in­guishes them from out­comes that we would want. When I was a small child I was cer­tain that I would never want to move away from home. Then I grew up, changed my mind, and moved away from home. It is im­por­tant that I was able to do some­thing which a past ver­sion of my­self would be hor­rified by. But this does not im­ply that there should be a gen­eral rule al­low­ing all such changes. Un­der­stand­ing which changes to your util­ity func­tion are good or bad is, as far as de­ci­sion the­ory is con­cerned, un­defined.

[jacob_cannell]

I hon­estly have a difficult time un­der­stand­ing the peo­ple (such as your “AI al­ign­ment re­searchers and other LWers, Mo­ral philoso­phers”) who ac­tu­ally be­lieve in Mo­ral­ity with a cap­i­tal M. I be­lieve they are mis­guided at best, po­ten­tially dan­ger­ous at worst.

I hadn’t heard of the Sta­tus Game book you quote, but for a long time now it’s seemed ob­vi­ous to me that there is no ob­jec­tive true Mo­ral­ity, it’s purely a cul­tural con­struct, and mostly a sta­tus game. Any deep read­ing of his­tory, cul­tures, and re­li­gions, leads one to this con­clu­sion.

Hu­mans have com­plex val­ues, and that is all.

We hu­mans co­op­er­ate and com­pete to op­ti­mize the uni­verse ac­cord­ing to those val­ues, as we always have, as our posthu­man de­scen­dants will, even with­out fully un­der­stand­ing them.

[Ape in the coat]

I think you are mi­s­un­der­stand­ing what Wei_Dai meant by “AI al­ign­ment re­searchers and other LWers, Mo­ral philoso­phers” per­spec­tive on moral­ity. It’s not about cap­i­tal let­ters or “ob­jec­tivity” of our moral­ity. It’s about that ex­act fact that hu­mans have com­plex val­ues and whether we can un­der­stand them and trans­late them into one course of ac­tion ac­cord­ing to which we are go­ing to op­ti­mize the uni­verse.

Ba­si­cally, as I un­der­stand it, the differ­ence is be­tween peo­ple who try to re­solve the con­filcts be­tween their differ­ent val­ues and gen­er­ally think about them as an ap­prox­i­ma­tion of some co­her­ent util­ity func­tion, and those who don’t.

[jacob_cannell]

It’s about that ex­act fact that hu­mans have com­plex val­ues and whether we can un­der­stand them and trans­late them into one course of ac­tion ac­cord­ing to which we are go­ing to op­ti­mize the uni­verse.

If we agree hu­mans have com­plex sub­jec­tive val­ues, then op­ti­miz­ing group de­ci­sions (for a mix of agents with differ­ent util­ity func­tions) is firmly a ques­tion for eco­nomic mechanism de­sign—which is already a rea­son­ably ma­ture field.

[JenniferRM]

A prob­lem here, how­ever, is the My­er­son–Sat­terth­waite re­sult which sug­gests that auc­tion run­ners, to en­able clean and helpful auc­tions for oth­ers, risk be­ing hurt when they ex­press and seek their own true prefer­ences, or (if they take no such risks) be­come bad auc­tion­eers for oth­ers.

The thing that seems like it might just be True here is that Good Gover­nance re­quires per­sonal sac­ri­fice by lead­ers, which I mostly don’t ex­pect to hap­pen, given nor­mal hu­man lead­ers, un­less those lead­ers are mo­ti­vated by, es­sen­tially: “al­tru­is­tic” “moral sen­ti­ment”.

It could be that I’m mi­s­un­der­stand­ing some part of the eco­nomics or the an­thro­pol­ogy or some such?

But it looks to me like if some­one says that there is no such thing as moral sen­ti­ment, it im­plies that they them­selves do not have such sen­ti­ments, and so per­haps those spe­cific peo­ple should not be given power or au­thor­ity or re­spect in so­cial pro­cesses that are vol­un­tary, uni­ver­sal, benev­olent, and the­o­ret­i­cally co­her­ent.

The rea­son­able­ness of this con­clu­sion goes some way to ex­plain to me how there is so much “so­cial sig­nal­ing” and also goes to ex­plain­ing why so much of this sig­nal­ing is fake garbage trans­mit­ted into the so­cial en­vi­ron­ment by power-hun­gry psy­chos.

[Ape in the coat]

Well, that’s one way to do it. With it’s own ter­rible con­se­quences, but lets not fo­cus on them for now.

What’s more im­por­tant is that this solu­tion is very gen­eral, while all hu­man val­ues be­long to the same cluster. So there may be more prefer­able, more hu­man-spe­cific solu­tion for the prob­lem.

[Shmi]

To re­post my com­ment from a cou­ple of weeks back, which seems to say roughly the same thing, not as well:

I don’t be­lieve al­ign­ment is pos­si­ble. Hu­mans are not al­igned with other hu­mans, and the only thing that pre­vents an im­me­di­ate apoc­a­lypse is the lack of re­cur­sive self-im­prove­ment on short timescales. Cer­tainly groups of hu­mans hap­pily de­stroy other groups of hu­mans, and of­ten de­stroy them­selves in the pro­cess of max­i­miz­ing some­thing like the num­ber of stat­ues. Best we can hope for that what­ever takes over the planet af­ter meat­bags are gone has some of the same goals that the more en­light­ened meat­bags had, where “en­light­ened” is a very in­di­vi­d­ual defi­ni­tion. Maybe it is a thriv­ing and di­verse Galac­tic civ­i­liza­tion, maybe it is the word of God spread to the stars, maybe it is liv­ing quietly on this planet in har­mony with the na­ture. There is no sin­gle or even shared vi­sion of the fu­ture that can be de­scribed as “al­igned” by most hu­mans.

[Charlie Steiner]

Do you think there are changes to the cur­rent world that would be “al­igned”? (E.g. delet­ing covid) Then we could end up with a world that is bet­ter than our cur­rent one, even with­out need­ing all hu­mans to agree on what’s best.

Another op­tion: why not just do ev­ery­thing at once? Have some peo­ple liv­ing in a di­verse Galac­tic civ­i­liza­tion, other peo­ple spread­ing the word of god, and other peo­ple liv­ing in har­mony with na­ture, and ev­ery­one con­tribut­ing a lit­tle to ev­ery­one else’s goals? Yes, in prin­ci­ple peo­ple can have differ­ent val­ues such that this fu­ture sounds ter­rible to ev­ery­one—but in re­al­ity it seems more like peo­ple would pre­fer this to our cur­rent world, but might merely feel like they were miss­ing out rel­a­tive to their own vi­sion of perfec­tion.

[Ratios]

I have also made a similar com­ment a few weeks ago, In fact, this point seems to me so triv­ial yet cor­ro­sive that I find it out­right bizarre it’s not be­ing tack­led/​taken se­ri­ously by the AI al­ign­ment com­mu­nity.

[WalterL]

I’m not sure what you mean by ‘as­tro­nom­i­cal waste or as­tro­nom­i­cal suffer­ing’. Like, you are writ­ing that ev­ery­thing for­ever is sta­tus games, ok, sure, but then you can’t turn around and ap­peal to a uni­ver­sal con­cept of suffer­ing/​waste, right?

What­ever you are wor­ried about is just like Gandhi wor­ry­ing about be­ing too con­cerned with cat­tle, plus x years, yeah? And even if you’ve lucked into a non sta­tus games moral­ity such that you can per­ceive ‘Gen­uine Waste’ or what have you...surely by your own logic, we who are read­ing this are in­ca­pable of un­der­stand­ing, aside from in terms of sta­tus games.

[Wei Dai]

I’m sug­gest­ing that maybe some of us lucked into a sta­tus game where we use “rea­son” and “de­liber­a­tion” and “do­ing philos­o­phy” to com­pete for sta­tus, and that some­how “do­ing philos­o­phy” etc. is a real thing that even­tu­ally leads to real an­swers about what val­ues we should have (which may or may not de­pend on who we are). Of course I’m far from cer­tain about this, but at least part of me wants to act as if it’s true, be­cause what other choice does it have?

[null]

The al­ter­na­tive is ego­ism. To the ex­tent that we are al­lies, I’d be happy if you adopted it.

[Wei Dai]

I don’t think that’s a vi­able al­ter­na­tive, given that I don’t be­lieve that ego­ism is cer­tainly right (surely the right way to treat moral un­cer­tainty can’t be to just pick some­thing and “adopt it”?), plus I don’t even know how to adopt ego­ism if I wanted to:

• https://​​www.less­wrong.com/​​posts/​​Nz62ZurRkGPigAxMK/​​where-do-self­ish-val­ues-come-from

• https://​​www.less­wrong.com/​​posts/​​c73kPDr8pZGdZSe3q/​​solv­ing-self­ish­ness-for-udt (which doesn’t re­ally solve the prob­lem de­spite the ti­tle)

[TekhneMakre]

So on the one hand you have val­ues that are eas­ily, triv­ially com­pat­i­ble, such as “I want to spend 1000 years climb­ing the moun­tains of Mars” or “I want to host blood-sports with my un­co­erced friends with the holodeck safety on”.

On the other hand you have in­sol­u­ble, or at least ap­par­ently in­sol­u­ble, con­flicts: B wants to tor­ture peo­ple, C wants there to be no tor­ture any­where at all. C wants to mon­i­tor ev­ery­one ev­ery­where for­ever to check that they aren’t tor­tur­ing any­one or plot­ting to tor­ture any­one, D wants pri­vacy. E and F both want to be the best in the uni­verse at quan­tum soc­cer, even if they have to kneecap ev­ery­one else to get that. Etc.

It’s sim­ply false that you can just put peo­ple in the throne as em­peror of the uni­verse, and they’ll justly com­pro­mise about all con­flicts. Or even do any­thing re­motely like that.

How many peo­ple have con­flic­tual val­ues that they, effec­tively, value lex­i­co­graph­i­cally more than their other val­ues? Does de­ci­sion the­ory im­ply that com­pro­mise will be cho­sen by suffi­ciently well-in­formed agents who do not have lex­i­co­graph­i­cally val­ued con­flic­tual val­ues?

[Vladimir_Nesov]

I’m lean­ing to­wards the more am­bi­tious ver­sion of the pro­ject of AI al­ign­ment be­ing about cor­rigible anti-good­hart­ing, with the AI op­ti­miz­ing to­wards good tra­jec­to­ries within scope of rel­a­tively well-un­der­stood val­ues, pre­vent­ing overop­ti­mized weird/​con­tro­ver­sial situ­a­tions, even at the cost of as­tro­nom­i­cal waste. Ab­sence of x-risks, in­clud­ing AI risks, is gen­er­ally good. Within this en­vi­ron­ment, the civ­i­liza­tion might be able to even­tu­ally work out more about val­ues, ex­pand­ing the scope of their defi­ni­tion and thus al­low­ing stronger op­ti­miza­tion. Here cor­rigi­bil­ity is in part about con­tinu­ally pick­ing up the val­ues and their im­plied scope from the pre­dic­tions of how they would’ve been worked out some time in the fu­ture.

[Wei Dai]

I’m lean­ing to­wards the more am­bi­tious ver­sion of the pro­ject of AI al­ign­ment be­ing about cor­rigible anti-good­hart­ing, with the AI op­ti­miz­ing to­wards good tra­jec­to­ries within scope of rel­a­tively well-un­der­stood values

Please say more about this? What are some ex­am­ples of “rel­a­tively well-un­der­stood val­ues”, and what kind of AI do you have in mind that can po­ten­tially safely op­ti­mize “to­wards good tra­jec­to­ries within scope” of these val­ues?

[Vladimir_Nesov]

My point is that the al­ign­ment (val­ues) part of AI al­ign­ment is least ur­gent/​rele­vant to the cur­rent AI risk crisis. It’s all about cor­rigi­bil­ity and anti-good­hart­ing. Cor­rigi­bil­ity is hope for even­tual al­ign­ment, and anti-good­hart­ing makes in­ad­e­quacy of cur­rent al­ign­ment and im­perfect ro­bust­ness of cor­rigi­bil­ity less of a prob­lem. I gave the rele­vant ex­am­ple of rel­a­tively well-un­der­stood val­ues, prefer­ence for lower x-risks. Other val­ues are mostly rele­vant in how their un­der­stand­ing de­ter­mines the bound­ary of anti-good­hart­ing, what counts as not too weird for them to ap­ply, not in what they say is bet­ter. If anti-good­hart­ing holds (too weird and too high im­pact situ­a­tions are not pur­sued in plan­ning and pos­si­bly ac­tively dis­cour­aged), and some sort of long re­flec­tion is still go­ing on, cur­rent al­ign­ment (de­tails of what the val­ues-in-AI pre­fer, as op­posed to what they can make sense of) doesn’t mat­ter in the long run.

I in­clude main­tain­ing a well-de­signed long re­flec­tion some­where into cor­rigi­bil­ity, for with­out it there is no hope for even­tual al­ign­ment, so a de­ci­sion the­o­retic agent that has long re­flec­tion within its prefer­ence is cor­rigible in this sense. Its cor­rigi­bil­ity de­pends on fol­low­ing a good de­ci­sion the­ory, so that there ac­tu­ally ex­ists a way for the long re­flec­tion to de­ter­mine its prefer­ence so that it causes the agent to act as the long re­flec­tion wishes. But be­ing an op­ti­mizer it’s hor­ribly not anti-good­hart­ing, so can’t be stopped and prob­a­bly eats ev­ery­thing else.

An AI with anti-good­hart­ing turned to the max is the same as AI with its stop but­ton pressed. An AI with min­i­mal anti-good­hart­ing is an op­ti­mizer, AI risk in­car­nate. Stronger anti-good­hart­ing is a main­te­nance mode, op­por­tu­nity for fun­da­men­tal change, weaker anti-good­hart­ing makes use of more de­vel­oped val­ues to ac­tu­ally do the things. So a way to con­trol the level of anti-good­hart­ing in an AI is a cor­rigi­bil­ity tech­nique. The two con­cepts work well with each other.

[Wei Dai]

This seems in­ter­est­ing and novel to me, but (of course) I’m still skep­ti­cal.

I gave the rele­vant ex­am­ple of rel­a­tively well-un­der­stood val­ues, prefer­ence for lower x-risks.

Prefer­ence for lower x-risk doesn’t seem “well-un­der­stood” to me, if we in­clude in “x-risk” things like value drift/​cor­rup­tion, pre­ma­ture value lock-in, and other highly con­se­quen­tial AI-en­abled de­ci­sions (po­ten­tial ex­is­ten­tial mis­takes) that de­pend on hard philo­soph­i­cal ques­tions. I gave some spe­cific ex­am­ples in this re­cent com­ment. What do you think about the prob­lems on that list? (Do you agree that they are se­ri­ous prob­lems, and if so how do you en­vi­sion them be­ing solved or pre­vented in your sce­nario?)

[Ratios]

The fact that AI al­ign­ment re­search is 99% about con­trol, and 1% (maybe less?) about metaethics (In the con­text of how do we even ag­gre­gate the util­ity func­tion of all hu­man­ity) hints at what is re­ally go­ing on, and that’s enough said.

[Daniel Kokotajlo]

Have you heard about CEV and Fun The­ory? In an ear­lier, more op­ti­mistic time, this was in­deed a ma­jor fo­cus. What changed is we be­came more pes­simistic and de­cided to fo­cus more on first things first—if you can’t con­trol the AI at all, it doesn’t mat­ter what metaethics re­search you’ve done. Also, the longter­mist EA com­mu­nity still thinks a lot about metaethics rel­a­tive to liter­ally ev­ery other com­mu­nity I know of, on par with and per­haps slightly more than my philos­o­phy grad stu­dent friends. (That’s my take at any rate, I haven’t been around that long.)

[Ratios]

CEV was writ­ten in 2004, fun the­ory 13 years ago. I couldn’t find any re­cent MIRI pa­per that was about metaethics (Granted I haven’t gone through all of them). The metaethics ques­tion is im­por­tant just as much as the con­trol ques­tion for any util­i­tar­ian (What good will it be to con­trol an AI only for it to be al­igned with some re­ally bad val­ues, an AI-con­trol­led by a sadis­tic so­ciopath is in­finitely worse than a pa­per-clip-max­i­mizer). Yet all the re­search is fo­cused on con­trol, and it’s very hard not to be cyn­i­cal about it. If some peo­ple be­lieve they are cre­at­ing a god, it’s self­ishly pru­dent to make sure you’re the one hold­ing the reigns to this god. I don’t get why hav­ing some blind trust in the benev­olence of Peter Thiel (who fi­nances this) or other peo­ple who will sud­denly have godly pow­ers to care for all hu­man­ity seems naive with all we know about how power cor­rupts and how com­pet­i­tive and self­ish peo­ple are. Most peo­ple are not util­i­tar­i­ans, so as a quasi-util­i­tar­ian I’m pretty ter­rified of what kind of world will be cre­ated with an AI-con­trol­led by the typ­i­cal non-util­i­tar­ian per­son.

[Daniel Kokotajlo]

My claim was not that MIRI is do­ing lots of work on metaethics. As far as I know they are fo­cused on the con­trol/​al­ign­ment prob­lem. This is not be­cause they think it’s the only prob­lem that needs solv­ing; it’s just the most dire, the biggest bot­tle­neck, in their opinion.

You may be in­ter­ested to know that I share your con­cerns about what hap­pens af­ter (if) we suc­ceed at solv­ing al­ign­ment. So do many other peo­ple in the com­mu­nity, I as­sure you. (Though I agree on the mar­gin more quiet aware­ness-rais­ing about this would plau­si­bly be good.)

[Mitchell_Porter]

http://​​www.metaeth­i­cal.ai is the state of the art as far as I’m con­cerned…

[null]

If with “moral­ity” you mean moral re­al­ism, then yes, I agree that it is scary.
I’m most scared by the ap­par­ent as­sump­tion that we have solved the hu­man al­ign­ment prob­lem.
Look­ing at his­tory, I don’t feel like our cur­rent situ­a­tion of rel­a­tive peace is very sta­ble.
My im­pres­sion is that “good” be­hav­ior is largely de­pen­dent on in­cen­tives, and so is the very defi­ni­tion of “good”.
Per­haps mar­kets are one of the more suc­cess­ful tools of cre­at­ing al­igned be­havi­our in hu­mans, but even in that case it only seems to work if the pow­ers of the mar­ket par­ti­ci­pants are bal­anced, which is not a lux­ury we have in al­ign­ment work.

[Ben123]

You could read the sta­tus game ar­gu­ment the op­po­site way: Maybe sta­tus seek­ing causes moral be­liefs with­out jus­tify­ing them, in the same way that it can dis­tort our fac­tual be­liefs about the world. If we can de­bunk moral be­liefs by find­ing them to be only sta­tus-mo­ti­vated, the sta­tus ex­pla­na­tion can ac­tu­ally as­sist ra­tio­nal re­flec­tion on moral­ity.

Also the quote from The Sta­tus Game con­flates purely moral be­liefs and fac­tual be­liefs in a way that IMO weak­ens its ar­gu­ment. It’s not clear that many of the ex­am­ples of crazy value sys­tems would sur­vive full log­i­cal and em­piri­cal in­for­ma­tion.

[Wei Dai]

The point I was try­ing to make with the quote is that many peo­ple are not mo­ti­vated to do “ra­tio­nal re­flec­tion on moral­ity” or ex­am­ine their value sys­tems to see if they would “sur­vive full log­i­cal and em­piri­cal in­for­ma­tion”. In fact they’re mo­ti­vated to do the op­po­site, to pro­tect their value sys­tems against such re­flec­tion/​ex­am­i­na­tion. I’m wor­ried that al­ign­ment re­searchers are not wor­ried enough that if an al­ign­ment scheme causes the AI to just “do what the user wants”, that could cause a lock-in of crazy value sys­tems that wouldn’t sur­vive full log­i­cal and em­piri­cal in­for­ma­tion.

[Gunnar_Zarncke]

There is no unique eu­topia.

Sen­tient be­ings that col­lab­o­rate out­com­pete ones that don’t (not con­sid­er­ing here, in­ner com­pe­ti­tion in a sin­gle­ton). Col­lab­o­ra­tion means that in­ter­ests be­tween be­ings are traded/​com­pro­mised. Bet­ter col­lab­o­ra­tion meth­ods have a higher chance to win. We see this over the course of his­tory. This is a messy evolu­tion­ary pro­cess. But I think there is a chance that this pro­cess it­self can be im­proved e.g. with FAI. Think of an in­ter­ac­tive “AlphaValue” that does Monte-Carlo Tree Search over col­lab­o­ra­tion op­por­tu­ni­ties. It will not con­verge on a unique best CEV but re­sult in one of many pos­si­ble eu­topias.

[c.trout]

I don’t fol­low the rea­son­ing. How do you get from “most peo­ple’s moral be­havi­our is ex­plain­able in terms of them ‘play­ing’ a sta­tus game” to “solv­ing (some ver­sions of) the al­ign­ment prob­lem prob­a­bly won’t be enough to en­sure a fu­ture that’s free from as­tro­nom­i­cal waste or as­tro­nom­i­cal suffer­ing”?

More de­tails:
Re­gard­ing the quote from The Sta­tus Game: I have not read the book, so I’m not sure what the in­tended mes­sage is but this sounds like some sort of un­war­ranted pes­simism about ppl’s moral stand­ing (some­thing like a claim like “the vast ma­jor­ity of ppl are morally ugly in this way”). There is all the differ­ence be­tween be­ing able to ex­plain most ppl’s be­havi­our in terms of “play­ing a sta­tus game” (in some sense of the word “play­ing”), and claiming that most ppl’s con­scious mo­ti­va­tion to act morally is to win at the sta­tus game. The lat­ter claim could plau­si­bly war­rant the pes­simism; the former, not. But I don’t see an ar­gu­ment for the lat­ter. Why is the former claim not ev­i­dence for ugli­ness? For the same rea­son that the claim “a mother’s love for their child is ge­net­i­cally ‘hard-wired’” is not ev­i­dence that a mother’s love for their child is ugly (or fake, not gen­uine etc). Ex­plain­ing the un­der­ly­ing causal mechanisms of a given moral be­havi­our is not (in gen­eral) enough to war­rant a given moral judg­ment. (If in­stead the book is ar­gu­ing for some sort of meta-eth­i­cal anti-re­al­ism, well, then the dis­cus­sion needs to be much longer...)

Re­gard­ing your fear about moral­ity: is the worry that if we just ag­gre­gated ev­ery­one’s val­ues we would get a lock-in to some sort of “ugly” sta­tus game? Again, we need more de­tails on its pro­posed im­ple­men­ta­tion be­fore we can judge whether its ugly (some­thing to be scared of).

But also, why are we as­sum­ing some sort of ag­gre­ga­tion of first-or­der hu­man value prefer­ences (no mat­ter the method of ag­gre­ga­tion)? As­sum­ing we’re talk­ing about AGI (and not CAIS), I always thought it strange to think we need to make sure it shares our own ideal­ized prefer­ences, as op­posed to merely the prefer­ences we would hope for in, say, a benev­olent god or some­thing. I don’t see any a pri­ori rea­son to be­lieve that the prefer­ences/​goals of a benev­olent shep­herd are likely to be shared with/​or strongly al­igned with those of the shep­herd’s flock (no mat­ter how you ag­gre­gate the prefer­ences/​goals of the flock). (I sup­pose it de­pends on the na­ture of the species we’re talk­ing about, but whether it’s sheep or hu­mans, I main­tain my skep­ti­cism). In any case, I agree with you that a lot more meta-ethics needs to be dis­cussed in the al­ign­ment re­search com­mu­nity.

[ThomasMore]

Great post, thanks! Wide­spread value plu­ral­ism a la ‘well that’s just, like, your opinion man’ is now a fea­ture of mod­ern life. Here are a pair of re­sponses from poli­ti­cal philos­o­phy which may be of some in­ter­est

(1) Rawls/​Thin Liberal Ap­proach. Whilst we may not be able to agree on what ‘the good life’ is, we can at least agree on a ba­sic sys­tem which en­sures all par­ti­ci­pants can pur­sue their own idea of the good life. So,(1) Pro­tect a list of poli­ti­cal liber­ties and free­doms and (2) de­gree of eco­nomic lev­el­ling. Beyond that, it is up to the in­di­vi­d­ual what con­cept of the good they pur­sue. Scott Alexan­der’s Archipelago is ar­guably a ver­sion of this the­ory, albeit with a plu­ral­ity of com­mu­ni­ties rather than a sin­gle state. Note it is ‘thin’ but not nonex­is­tent—ob­vi­ously cer­tain con­cepts of the good, such as ‘kil­ling/​en­slav­ing ev­ery­one for my God’, are in­com­pat­i­ble and ex­cluded.

(2) Nuss­baum ‘Ca­pac­ity’ Ap­proach. Bit like Liberal+ ap­proach. You take the liberal ap­proach then beef it up by adding some more re­quire­ments: you need to pro­tect the ca­pac­ity of peo­ple to achieve wellbe­ing. Ba­si­cally—pro­tect life, en­vi­ron­ment, health/​bod­ily in­tegrity, ed­u­ca­tion (sci­en­tific & cre­ative), prac­ti­cal rea­son, be­ing able to play, hold prop­erty, form emo­tional and so­cial at­tach­ments. The main differ­ence with (2) is that it is a thicker con­cep­tion of the ‘good life’ - it will deny var­i­ous tra­di­tional forms of life on the ba­sis they do not ed­u­cate their chil­dren or give them crit­i­cal think­ing skills. Hence, Nuss­baum cham­pi­ons the no­tion of ‘uni­ver­sal val­ues.’

Go­ing from (1) to (2) de­pends on how com­fortable you are with an ob­jec­tive no­tion of flour­ish­ing. IMO it’s not to­tally im­plau­si­ble given com­mon­al­ities across cul­tures of val­ues (which Nuss­baum points out—moral rel­a­tivism is of­ten ex­ag­ger­ated) and var­i­ous as­pects of hu­man ex­pe­rience.

[Tee Canker]

Don’t you need AI to go through the many mil­lions of ex­pe­riences that it might take to de­velop a good moral­ity strat­egy?

I’m en­tranced by Jor­dan Peter­son’s de­scrip­tions, which seem to light up the evolu­tion­ary path of moral­ity for hu­mans. Shouldn’t AI be set up to try to grind through the same progress?

[Andrew McKnight]

I think the main thing you’re miss­ing here is that an AI is not gen­er­ally go­ing to share com­mon learn­ing fa­cil­ities with hu­mans. An AI grow­ing up as a hu­man will make it wildly differ­ent from a nor­mal hu­man be­cause they aren’t built pre­cisely to learn from those ex­pe­riences the way a hu­man does.

[Flaglandbase]

What’s truly scary is how much the be­liefs and opinions of nor­mal peo­ple make them seem like aliens to me.

[anon312]

I find the para­graph be­gin­ning with these two sen­tences, its ex­am­ples mis­lead­ing and un­con­vinc­ing in the point about moral dis­agree­ment across time it tries to make:

Such ‘facts’ also change across time. We don’t have to travel back far to dis­cover moral su­per­stars hold­ing moral views that would de­stroy them to­day.

I shall try to ex­plain why, be­cause such ev­i­dence seemed per­sua­sive to me be­fore I thought about it more; I made this ac­count just for this com­ment af­ter be­ing a lurker for a while—I have found your pre­vi­ous posts about moral un­cer­tainty quite in­ter­est­ing.

When com­par­ing moral views across time to see if there is dis­agree­ment or agree­ment, we should see if there is an un­der­ly­ing prin­ci­ple that is stated by the per­son or one that can be in­ferred, which is differ­ent from the spe­cific con­tex­tual judge­ment.

The ap­pli­ca­tion of the same prin­ci­ples across time to var­i­ous con­crete mat­ters will widely vary de­pend­ing on the con­text and in­for­ma­tion of the per­son ap­ply­ing them, that per­son’s ca­pa­bil­ities. There­fore when seek­ing to de­ter­mine moral agree­ment or dis­agree­ment with a spe­cific judge­ment, one should not give ex­am­ples of spe­cific prac­ti­cal moral judge­ments with­out also giv­ing in­for­ma­tion about the un­der­ly­ing mo­ti­va­tion and fac­tual view of the world as­so­ci­ated with it.

One can see how much util­i­tar­i­ans with the same the­ory can di­verge on what to do. But they dis­agree some­times not in moral the­ory, but on things that are un-am­bigu­ously em­piri­cal mat­ters. So peo­ple can dis­agree on a spe­cific moral judg­ment with­out that dis­agree­ment be­ing of a moral na­ture. Be­cause peo­ple across time differ quite a lot in their non-moral fac­tual un­der­stand­ing of world, one would need to do re­search and in­ter­pre­ta­tion from case to case to un­cover the ac­tual prin­ci­ples at play in var­i­ous moral judge­ments which seem strange or dis­agree­able to us. Without try­ing to con­sider the moral rules or prin­ci­ples be­ing ap­plied, and the un­der­stand­ing of the world they are be­ing ap­plied to, how can we de­ter­mine if we dis­agree with some­one about a moral fact, or an­other fact about the world?

To give a sim­ple ex­am­ple about how moral judge­ments and fac­tual judge­ments are re­lated: it is morally easy for us to see that kil­ling witches was wrong, be­cause there were no witches. The com­mu­ni­ties that burned witches may have differed morally from us, but there is enough moral agree­ment be­tween us and them that if they had not be­lieved in witches or those spe­cific false in­stances of witches, they would not have burned them (even when there was ly­ing about witches in­volved in his­tor­i­cal con­texts of witch burn­ing, that ly­ing took place in a com­mu­nity that be­lieved in witches and wanted to pun­ish them ac­cord­ingly).

It would not make sense to say some­thing like

But there’s about as much sense in blam­ing Gandhi for not shar­ing our mod­ern, Western views on race as there is in blam­ing the Vik­ings for not hav­ing Net­flix.

about the com­mu­ni­ties that burned witches. I think we can cor­rectly say they were ob­jec­tively mis­taken to burn those witches. I’m not very sure about much in­volv­ing moral­ity, but that is the sort of mod­est ‘moral re­al­ism’ I am con­fi­dent hav­ing when judg­ing the his­toric moral de­ci­sions of past com­mu­ni­ties and in­di­vi­d­u­als (I am not sure about the crite­ria for ‘blam­ing’ but I am con­fi­dent we can say they were ob­jec­tively wrong to do that, like about any other fac­tual mat­ter). The loosely held shared moral prin­ci­ple would be some­thing like ‘do not kill peo­ple for en­tirely mis­taken rea­sons’. I am not say­ing that will be the case, lo­cat­ing a shared moral prin­ci­ple, for ev­ery past moral de­ci­sion we might dis­agree with. But for more com­pli­cated cases we re­ally do have to re­search be­fore de­ter­min­ing if there is a gen­uinely moral dis­agree­ment IMO, rather than just stat­ing var­i­ous spe­cific judge­ments we seem to dis­agree with.

So in the case of Marie Stopes and Gandhi, among other ex­am­ples, I am un­able to tell if I morally dis­agree with them based on the limited in­for­ma­tion given in the ex­cerpt. I would need to know more about their view of the facts of the mat­ters men­tioned and try to un­der­stand and in­fer from what they have shared el­se­where, the moral views be­ing ap­plied (it is es­pe­cially true in the case of in­no­va­tive moral ac­tivists, that they will ar­tic­u­late their prin­ci­ples, so it is odd that the au­thor does not tell us them).

I hope I have been con­vinc­ing and rea­son­able about why we should be wary of tel­ling what some­one’s un­der­ly­ing moral view is and our agree­ment or dis­agree­ment with it from limited ex­am­ples of the sort the au­thor gives here; this is even more true for pre-mod­ern cul­tures with very differ­ent un­der­stand­ings of the world, like the Mala­gasy and their taboos. How tightly wound im­plicit moral views and em­piri­cal views can be is clear with the case of com­mu­nism, an­other ex­am­ple given. There is enough moral feel­ing, be­lief in com­mon that many un­usu­ally-bad-to-us cases of moral be­hav­ior are also tied to a very differ­ent em­piri­cal view of the world.

One fi­nal note, re­gard­ing sta­tus games and moral­ity. I think the fact that many peo­ple are urged into be­hav­ior con­sid­ered moral by the ex­pec­ta­tions of oth­ers who find moral be­hav­ior praise­wor­thy, shows that the con­cept of moral­ity can be ob­jec­tively ap­plied in sce­nar­ios well out­side spe­cific peo­ple hav­ing an au­tonomous in­trin­si­cally moral mo­ti­va­tion. Not ev­ery­one wants to act morally even if they rec­og­nize an ob­jec­tive moral ideal, but of­ten in­stru­men­tally is driven to en­gage in acts con­sid­ered moral, or meet­ing the re­spectable moral min­i­mum at least, by so­ciety for their own amoral rea­sons.

We can even imag­ine a world where no one acts for in­trin­si­cally moral rea­sons, but peo­ple still some­times shape their ac­tions around a con­cept of ob­jec­tive moral­ity that arises be­cause peo­ple are able to rec­og­nize their own self­ish in­ter­ests and rec­og­nize, want to re­ward some­one ‘al­tru­is­ti­cally’ con­tribut­ing to their in­ter­ests or at least re­spect­ing them enough not to hurt them. I don’t think that world is ours, but I do think in our world many peo­ple do not have pos­i­tive moral mo­ti­va­tions for much of their be­hav­ior and moral re­form­ers, saint-types are known as such for hav­ing un­usu­ally strong in­trin­sic moral mo­ti­va­tions and for driv­ing peo­ple to fur­ther live up to the moral ideals they merely su­perfi­cially meet for in­stru­men­tal rea­sons most of the time.

One can imag­ine ‘moral saints’ ne­go­ti­at­ing among var­i­ous peo­ple’s self­ish in­ter­ests to find or push peo­ple into the pos­i­tive sum out­come they them­selves could not self­ishly have got­ten into. I do not think pri­mar­ily moral agents and ‘en­light­ened’ self­ish ones are nec­es­sar­ily op­posed at all (this sort of ne­go­ti­a­tion is how one can pic­ture a broader hu­man co­op­er­a­tion /​ al­ign­ment in the con­text of pow­er­ful moral hu­man agents, in re­la­tion to the power of an al­igned ar­tifi­cial in­tel­li­gence).

The de­sire to ap­pear more moral than one re­ally is adds a lot of treach­ery to moral dis­course in all com­mu­ni­ties; moral the­o­riz­ing can offend peo­ple through in­trud­ing on illu­sions about how self­ish they are and this trick­i­ness is surely one rea­son for less progress on clar­ify­ing moral ob­jec­tivity than there would be oth­er­wise. It seems likely that due to hu­man moral im­perfec­tion there is in­e­liminable sub­op­ti­mal­ity in our ac­tions and ac­tual de­sires rel­a­tive to all of our ideal moral con­cepts (even if they can gen­uinely par­tially mo­ti­vate us in a more moral di­rec­tion through their at­trac­tion as pure ideals), and that this shall be trans­ferred to any agents that we cre­ate to satisfy us. It will prob­a­bly be like Kant said, the learn­ing of hu­man val­ues by an AI:

Out of the crooked tim­ber of hu­man­ity, no straight thing was ever made

[Ape in the coat]

It seems that our moral­ity con­sists of two el­e­ments. First is bias, based on game the­o­ret­i­cal en­vi­ron­ment of our an­ces­tors. Hu­mans de­vel­oped com­plex feel­ings around ac­tivi­ties that pro­moted in­clu­sive ge­netic fit­ness and now we are in­trin­si­cally and au­then­ti­cally mo­ti­vated to do them for their own sake.

There is also a limited ca­pa­bil­ity for moral up­dates. That’s what we use to re­solve con­tra­dic­tions in our moral in­tu­itions. And that’s also what al­low us to per­suade our­selves that do­ing some sta­tus pro­mot­ing thing is ac­tu­ally moral. One the one hand, the fact that it’s so easy to change our ethics due to sta­tus rea­sons is kind of scary. On the other, whole abil­ity to morally up­date was prob­a­bly de­vel­oped ex­actly for this, so it’s kind of a mir­a­cle that we can use it differ­ently at all.

I don’t think that we can up­date into gen­uinely feel­ing that max­i­miz­ing pa­per­clips for its own sake is the right thing to do. All pos­si­ble hu­man minds oc­cupy only a small part of all pos­si­ble minds space. We can con­sider al­ign­ment to be somwhat solved if TAI guaran­tee us op­ti­miza­tion in the di­rec­tion of some neigh­bour­hood of our moral bias. How­ever, I think it’s pos­si­ble to do bet­ter and we do not need all hu­mans to be moral philoso­phers for that. It will be enough if TAI it­self is a perfect moral philoso­pher able to deduct our co­her­ent ex­trap­o­lated vo­li­tion and be­come an op­ti­miza­tion pro­cess in that di­rec­tion.

[romeostevensit]

You may not be in­ter­ested in mu­tu­ally ex­clu­sive com­pres­sion schemas, but mu­tu­ally ex­clu­sive com­pres­sion schemas are in­ter­ested in you. One nice thing is that given that the schemas use an ar­bi­trary key to hand­shake with there is hope that they can be con­vinced to all get on the same ar­bi­trary key with­out loss of use­ful struc­ture.

[deleted]

[deleted]

[Wei Dai]

Can you please say some­thing about how these videos are rele­vant to my post?

[Tee Canker]

Sorry, I’m not used to this par­tic­u­lar in­ter­face.

I gave com­ments and did my best.

Thank you for shar­ing!!!