Stuart_Armstrong

Apoca­lypse, corrupted

Re­search Agenda in re­verse: what *would* a solu­tion look like?

• Stuart_Armstrong 25 Jun 2019 12:27 UTC

  2 points

  in reply to: Vladimir_Nesov's comment on: Up­com­ing sta­bil­ity of values

  Yep. But I note that many peo­ple seem to value let­ting their val­ues drift some­what, so that needs to be taken into ac­count.

• Stuart_Armstrong 22 Jun 2019 8:37 UTC

  2 points

  in reply to: J_Thomas_Moros's comment on: To first or­der, moral re­al­ism and moral anti-re­al­ism are the same thing

  What I meant was this: as­sume that 3^^^3 dust specks on one per­son is worse that 50 years of tor­ture. As long as the dust specks sen­sa­tion is some­what ad­di­tive, that should be true. Now sup­pose you have to choose be­tween dust specks and tor­ture 3^^^3 times, one for each per­son (“so, do we tor­ture in­di­vi­d­ual 27602, or one dust speck on ev­ery­one? Now, same ques­tion for 27602....).

  Then always choos­ing dust specks is worse, for ev­ery­one, than always choos­ing tor­ture.

  So the dust-speck de­ci­sion be­comes worse and worse, the more of­ten you ex­pect to en­counter it.

• Stuart_Armstrong 22 Jun 2019 8:36 UTC

  2 points

  AF

  in reply to: Charlie Steiner's comment on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  “hu­mans don’t have ac­tual prefer­ences so the AI is just go­ing to try to learn some­thing ad­e­quate.”

  Try some­thing like: hu­mans don’t have ac­tual con­sis­tent prefer­ences, so the AI is go­ing to try and find a good ap­prox­i­ma­tion that cov­ers all the con­tra­dic­tions and un­cer­tain­ties in hu­man prefer­ences.

• Stuart_Armstrong 22 Jun 2019 8:31 UTC

  4 points

  in reply to: Pattern's comment on: Up­com­ing sta­bil­ity of values

  In­for­ma­tion can (and should) change your be­havi­our, even if it doesn’t change your val­ues. Be­com­ing a par­ent should change your at­ti­tude to var­i­ous things whose pur­pose you didn’t see till then! And val­ues can pre­fer a va­ri­ety of ex­pe­riences, if we cash our bore­dom prop­erly.

  The prob­lem is that hu­mans mix in­for­ma­tion and val­ues to­gether in highly com­pli­cated, non-ra­tio­nal ways.

• Stuart_Armstrong 19 Jun 2019 14:16 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  I’d say that this prob­lem doesn’t be­long in sec­tion 2.3-2.4 (col­lect­ing and gen­er­al­is­ing prefer­ences), but in sec­tion 1.2 (sym­bol ground­ing, and es­pe­cially the web of con­no­ta­tions). That’s where these ques­tions should be solved, in my view.

  So yeah, I agree that stan­dard ma­chine learn­ing is not up to the task yet, at all.

  (as a minor aside, I’m also a bit un­sure how nec­es­sary it is to make par­tial prefer­ences to­tal be­fore com­bin­ing them; this may be un­nec­es­sary)

• Stuart_Armstrong 19 Jun 2019 14:12 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: For the past, in some ways only, we are moral degenerates

  Do these seem like things that could be “put in as a strong con­di­tional meta-prefer­ence” in your frame­work?

  Yes, very eas­ily.

  The main is­sue is whether these should count as an over­whelming meta-prefer­ence—one that over-weights all other con­sid­er­a­tions. And, cur­rently as I have things set up, the an­swer is no. I have no doubt that you feel strongly about po­ten­tially true moral re­al­ism. But I’m cer­tain that this strong feel­ing is not ab­surdly strong com­pared to other prefer­ences at other mo­ments in your life. So if we syn­the­sised your cur­rent prefer­ences, and 1. or 2. ended up be­ing true, then the moral re­al­ism would end up play­ing a large-but-not-dom­i­nat­ing role in your moral prefer­ences.

  I wouldn’t want to change that, be­cause what I’m aiming for is an ac­cu­rate syn­the­sis of your cur­rent prefer­ences, and your cur­rent prefer­ence for moral-re­al­ism-if-it’s-true is not, in prac­tice, dom­i­nat­ing your prefer­ences. If you wanted to en­sure the po­ten­tial dom­i­nance of moral re­al­ism, you’d have to put that di­rectly into the syn­the­sis pro­cess, as a global meta-prefer­ence (sec­tion 2.8 of the re­search agenda).

  But the whole dis­cus­sion feels a bit pe­cu­liar, to me. One prop­erty of moral re­al­ism that is of­ten as­sumed, is that it is, in some sense, ul­ti­mately con­vinc­ing—that all sys­tems of moral­ity (or all sys­tems de­rived from hu­mans) will con­verge to it. Yet when I said a “large-but-not-dom­i­nat­ing role in your moral prefer­ences”, I’m posit­ing that moral re­al­ism is true, but that we have a sys­tem of moral­ity - $U_H$ - that does not con­verge to it. I’m not re­ally grasp­ing how this could be pos­si­ble (you could ar­gue that the moral re­al­ism $U_R$ is some sort of acausal trade con­ver­gent func­tion, but that gives an in­stru­men­tal rea­son to fol­low $U_R$, not an ac­tual rea­son to have $U_R$; and I know that a moral sys­tem need not be a util­ity func­tion ^_^).

  So yes, I’m a bit con­fused by true-but-not-con­vinc­ing moral re­al­isms.

• Stuart_Armstrong 19 Jun 2019 14:08 UTC

  2 points

  in reply to: johnswentworth's comment on: Par­tial prefer­ences and models

  This differs, be­cause the z are as­sumed to be in a “stan­dard” range. There are situ­a­tions where ex­treme val­ues of z, if known and re­flected upon, would change the sign of the de­ci­sion (for ex­am­ple, what if your de­ci­sion is be­ing filmed, and there are billions be­ing bet upon your ul­ti­mate choice, by var­i­ous moral and im­moral groups?).

  But yeah, if you as­sume that the z are in that stan­dard range, then this looks a lot like con­sid­er­ing just a few nodes of a causal net.

• Stuart_Armstrong 19 Jun 2019 13:58 UTC

  4 points

  AF

  in reply to: Charlie Steiner's comment on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  One of the rea­sons I re­fer to syn­the­sis­ing (or con­struct­ing) the $U_H$, not learn­ing it.

• Stuart_Armstrong 18 Jun 2019 12:01 UTC

  4 points

  on: Rea­son isn’t magic

  Post was good, but I’d recom­mend adding an in­tro­duc­tory para­graph to the link on LessWrong.

• Stuart_Armstrong 18 Jun 2019 7:36 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: For the past, in some ways only, we are moral degenerates

  What scares me is the pos­si­bil­ity that moral anti-re­al­ism is false, but we build an AI un­der the as­sump­tion that it’s true

  One way of deal­ing with this, in part, is to figure out what would con­vince you that moral re­al­ism was true, and put that in as a strong con­di­tional meta-prefer­ence.

• Stuart_Armstrong 17 Jun 2019 18:21 UTC

  2 points

  in reply to: G Gordon Worley III's comment on: One-step hy­po­thet­i­cal preferences

  Valid point, though con­di­tional meta-prefer­ences are things I’ve already writ­ten about, and the is­sue of be­ing wrong now about what your own prefer­ences would be in the fu­ture, is also some­thing I’ve ad­dressed mul­ti­ple times in differ­ent forms. Your ex­am­ple is par­tic­u­larly crisp, though.

• Stuart_Armstrong 17 Jun 2019 18:19 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: For the past, in some ways only, we are moral degenerates

  I was talk­ing about “ex­tended fam­ily val­ues” in the sense of “it is good for fam­i­lies to stick to­gether and spend time with each other”; this prefer­ence can (and of­ten does) ap­ply to other fam­i­lies as well. I see no analogue for that with slav­ery.

  But yeah, you could ar­gue that racism can be a ter­mi­nal value, and that slave own­ers would de­velop it, as a jus­tifi­ca­tion for what might have started as an in­stru­men­tal value.

Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

• Stuart_Armstrong 17 Jun 2019 15:35 UTC

  4 points

  in reply to: G Gordon Worley III's comment on: One-step hy­po­thet­i­cal preferences

  I see the or­ange-ap­ple prefer­ence re­ver­sal as an­other ex­am­ple of con­di­tional prefer­ences.

Prefer­ence con­di­tional on cir­cum­stances and past prefer­ence satisfaction

• Stuart_Armstrong 17 Jun 2019 12:10 UTC

  5 points

  in reply to: rmoehn's comment on: Hu­mans can be as­signed any val­ues what­so­ever…

  Yep, ba­si­cally that. ^_^

• Stuart_Armstrong 17 Jun 2019 12:08 UTC

  2 points

  in reply to: Kaj_Sotala's comment on: For the past, in some ways only, we are moral degenerates

  I re­al­ize that you did ex­plic­itly define “de­gen­er­a­tion = mov­ing away from ours” for the drift­ing val­ues, but it feels weird to then also define “progress = mov­ing away from ours in a good way”

  If I de­com­posed a bit more, I’d say that we need to dis­t­in­guish the val­ues of oth­ers, and the state of the world, and whether things are mov­ing to­wards our val­ues, away from our val­ues, or just drift­ing.

  So “progress”, in the sense of my post, is com­posed of a) other peo­ple’s val­ues mov­ing to­wards our own, and b) the state of the world mov­ing more to­wards our own prefer­ences/​val­ues. “Mo­ral de­gen­er­a­tion”, on the other hand, is c) peo­ple’s val­ues drift away from our own.

  I see all three of these hap­pen­ing at once (along with, to some ex­tent “the state of the world mov­ing away from our val­ues”, which is an­other cat­e­gory), so that’s why we see both progress and de­gen­er­a­tion in the fu­ture.

• Stuart_Armstrong 17 Jun 2019 11:48 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: For the past, in some ways only, we are moral degenerates

  I think com­mu­nity and ex­tended fam­ily are (or were) ter­mi­nal val­ues, to some ex­tent at least (which doesn’t pre­clude them be­ing in­stru­men­tal val­ues also).