Vanessa Kosoy

• Vanessa Kosoy 13 Oct 2019 19:48 UTC

  2 points

  in reply to: shminux's comment on: What do the baby eaters tell us about ethics?

  Alright, now I fi­nally un­der­stand your claim. I still dis­agree with it: I think that your cyn­i­cism about hu­man mo­ti­va­tions is un­sup­ported by ev­i­dence. But, that’s not a de­bate I’m in­ter­ested to start atm. Thank you for ex­plain­ing your views.

• Vanessa Kosoy 13 Oct 2019 19:44 UTC

  4 points

  in reply to: Rob Bensinger's comment on: A sim­ple sketch of how re­al­ism be­came unpopular

  Let’s un­pack what it means to say that “re­friger­a­tors ex­ist”. From my (in­stru­men­tal­ist) per­spec­tive, it means that (i) I have a pre­dic­tive model of my per­cep­tion, to which I as­sign high cre­dence, and in which for each state of the en­vi­ron­ment I can say which re­friger­a­tors ex­ist (where “re­friger­a­tor” is just some sym­bol that makes sense in­side the model) and (ii) ac­cord­ing to my be­lief, the cur­rent state of the en­vi­ron­ment con­tains at least one re­friger­a­tor with high prob­a­bil­ity.

  My claim is not that quan­tum me­chan­ics proves re­al­ism is wrong. My claim is that in­stru­men­tal­ism is the cor­rect meta­physics re­gard­less, and once you ac­cept that, the Copen­hagen in­ter­pre­ta­tion seems quite satis­fac­tory. Although it is also true that if you try in­ter­pret­ing quan­tum me­chan­ics ac­cord­ing to suffi­ciently strong re­al­ist desider­ata you run into im­pos­si­bil­ity re­sults like the Kochen-Specker the­o­rem and the vi­o­la­tion of Bell’s in­equal­ity.

• Vanessa Kosoy 12 Oct 2019 12:29 UTC

  13 points

  on: A sim­ple sketch of how re­al­ism be­came unpopular

  This post was ed­u­ca­tional, how­ever, I want to push back against the im­plicit crit­i­cism of in­stru­men­tal­ism and the Copen­hagen in­ter­pre­ta­tion. The metaphilo­soph­i­cal po­si­tion I will use here is: to solve a philo­soph­i­cal ques­tion, we need to rephrase it as a ques­tion about AI de­sign (AFAIK the full credit for this ap­proach goes to Yud­kowsky). So, sup­pose I am build­ing an AI. Should the AI’s model of the world be (i) “about things that ex­ist in­de­pen­dently of the AI” or (ii) “about the sub­jec­tive per­cep­tions of the AI”? This de­pends on what kind of re­ward func­tion I want my AI to have.

  The stan­dard (I call it “per­cep­ti­ble”) type of re­ward func­tion in re­in­force­ment learn­ing only de­pends on the his­tory of ac­tions and ob­ser­va­tions. For such an AI the an­swer is (ii): it is im­por­tant the AI will cor­rectly pre­dict the con­se­quences of its ac­tions, but there is no sig­nifi­cance what­so­ever that the AI’s mod­els can be in­ter­preted as (i). Yes, these mod­els will still have in­trin­sic vari­ables cor­re­spond­ing to “un­ob­served ob­jects” in some sense, but there is no rea­son for these vari­ables to always have an un­am­bigu­ous “re­al­ist” in­ter­pre­ta­tion.

  Now sup­pose that the AI is ac­tu­ally de­signed to care about par­tic­u­lar ob­jects out­side it­self. Speci­fi­cally, as­sume the AI uses an in­stru­men­tal (or semi-in­stru­men­tal) re­ward func­tion. Such a func­tion might be speci­fied (or par­tially speci­fied) us­ing some par­tic­u­lar on­tol­ogy. Then, the AI is mod­el­ing the world as con­tain­ing cer­tain un­ob­served ob­jects, at least ap­prox­i­mately. The an­swer is now in be­tween (i) and (ii). The AI’s mod­els are about sub­jec­tive per­cep­tions of the AIs and also about some par­tic­u­lar type of things that ex­ist in­de­pen­dently of the AI, namely, those things that are nor­ma­tively im­por­tant to it. Com­ing back from AIs to hu­mans, we may con­clude that, what re­ally makes sense is our sub­jec­tive per­cep­tions + those ex­ter­nal ob­jects that we ac­tu­ally care about (e.g. other hu­mans). But, quarks only “ex­ist” in the sense that they are com­po­nents in a use­ful model we cre­ated.

  What does it tell us about the in­ter­pre­ta­tion of quan­tum me­chan­ics? Once again, con­sider an AI try­ing to dis­cover quan­tum me­chan­ics. From the AI’s per­spec­tive, what it’s look­ing for is a func­tion from ob­ser­va­tion his­to­ries to dis­tri­bu­tions over the next ob­ser­va­tion. How can we con­struct such a func­tion from the for­mal­ism of QM? Ob­vi­ously, us­ing Copen­hagen: each ob­ser­va­tion is a mea­sure­ment that causes the wave­func­tion of the en­vi­ron­ment to col­lapse. “But,” the fans of MWI will ob­ject “what about all those other Everett branches? They are still out there, right? They don’t ac­tu­ally van­ish?!” The point is, from the AI’s per­spec­tive the ques­tion is mean­ingless. The AI’s de­sign as­sumes the AI can record ev­ery­thing it ob­serves in its mem­o­ries, there­fore, once an ob­ser­va­tion is made, those Everett branches will never meet again in the AI’s life­time. “But,” an­other ob­jec­tion may go “what if some­one tem­pers with the mem­o­ries of the AI in a way that al­lows quan­tum in­terfer­ence be­tween the branches? Sure it is com­pletely im­prac­ti­cal for hu­mans, but it is the­o­ret­i­cally pos­si­ble, and might even be prac­ti­cally pos­si­ble for an AI run­ning on a quan­tum com­puter.” Alright, but tem­per­ing with the mem­ory of the agent is effec­tively de­stroy­ing the agent: it in­val­i­dates the fun­da­men­tal as­sump­tions of its rea­son­ing al­gorithm, and any rea­son­ing al­gorithm must make some fun­da­men­tal as­sump­tions of that sort. (The agent might still ac­cu­rately ac­count for the pos­si­bil­ity of its own de­struc­tion (see “The Death of the Agent and Kamikaze Strate­gies”), but prob­a­bly only with the help of ex­ter­nal knowl­edge.)

• Vanessa Kosoy 12 Oct 2019 9:50 UTC

  2 points

  in reply to: shminux's comment on: What do the baby eaters tell us about ethics?

  If I un­der­stood cor­rectly, your ob­jec­tion to Three Wor­lds Col­lide is (mostly?) de­scrip­tive rather than pre­scrip­tive: you think the story is un­re­al­is­tic, rather than dis­pute some nor­ma­tive po­si­tion that you be­lieve it defends. How­ever, de­pend­ing on the in­ter­pre­ta­tion of that maxim you for­mu­lated, it is (IMO) ei­ther fac­tu­ally wrong or en­tirely con­sis­tent with the story of Three Wor­lds Col­lide.

  Do you be­lieve real world hu­mans are “slow to act against the morals it finds ab­hor­rent”? If your an­swer is pos­i­tive, how do you ex­plain all (of­ten ex­tremely vi­o­lent) con­flicts over re­li­gion and poli­ti­cal ide­ol­ogy over the course of hu­man his­tory? What­ever ex­pla­na­tion you pro­pose to these con­flicts, what pre­vents it from ex­plain­ing the con­flict with the Baby-Eaters de­scribed in Three Wor­lds Col­lide? If your an­swer to the first ques­tion is nega­tive, how do you ex­plain the sur­vival of the hu­man species so far? What­ever ex­pla­na­tion you provide to this sur­vival, what pre­vents it from ex­plain­ing the con­tinued sur­vival of the hu­man species un­til the imag­i­nary fu­ture in the story?

• Vanessa Kosoy 11 Oct 2019 21:16 UTC

  27 points

  in reply to: TedSanders's comment on: How fea­si­ble is long-range fore­cast­ing?

  I agree with most of what you’re say­ing, but this part seems like giv­ing up way too eas­ily: “And even if you say, ok sure it de­pends, but like what’s the av­er­age an­swer—even then, the only the way to ar­rive at some un­bi­ased global sense of whether the fu­ture is pre­dictable is to come up with some way of enu­mer­at­ing and weigh­ing all pos­si­ble facts about the fu­ture uni­verse… which is an im­pos­si­ble prob­lem. So we’re left with the un­satis­fy­ing truth that the fu­ture is nei­ther pre­dictable or un­pre­dictable—it de­pends on which fea­tures of the fu­ture you are con­sid­er­ing.”

  The only way to say some­thing use­ful about this is liter­ally enu­mer­at­ing all pos­si­ble facts? Sounds need­lessly pes­simistic.

  On the con­trary, I think it could be tremen­dously in­ter­est­ing and use­ful to start build­ing some kind of cat­e­go­riza­tion of pre­dic­tion do­mains that al­lows say­ing some­thing about their re­spec­tive pre­dictabil­ity. Ob­vi­ously this is a hard prob­lem, ob­vi­ously the uni­verse is very com­plex and the cat­e­go­riza­tion will miss a lot of the in­tri­cacy, but the same is true about many other do­mains of knowl­edge (prob­a­bly, nearly all of them). De­spite the com­plex­ity of the uni­verse (i) we should keep look­ing for (ex­tremely) sim­plified mod­els that cap­ture a lot of what we ac­tu­ally care about (ii) hav­ing even an ex­tremely sim­plified model is of­ten much bet­ter than no model at all (iii) the model will keep evolv­ing over time (which is to say, it feels more like a po­ten­tial new sci­ence than a sin­gle prob­lem that can be stated and solved in a rel­a­tively short time frame).

• Vanessa Kosoy 11 Oct 2019 20:47 UTC

  2 points

  in reply to: shminux's comment on: What do the baby eaters tell us about ethics?

  I still don’t un­der­stand, is your claim de­scrip­tive or pre­scrip­tive?

  Nei­ther… Or maybe de­scrip­tive? I am sim­ply stat­ing the im­pli­ca­tion, not pre­scribing what to do.

  Then I don’t un­der­stand what you’re say­ing at all. If you are stat­ing an im­pli­ca­tion, then I don’t un­der­stand (i) what ex­actly is the premise (ii) what ex­actly is the con­clu­sion (iii) how is this im­pli­ca­tion vi­o­lated in the story Three World Col­lide.

  Yes, we do have plenty of laws, but no one goes out of their way to find and hunt down the vi­o­la­tors.

  So, your ar­gu­ment is (cor­rect me if I’m wrong): in the real world peo­ple only put that much effort into hunt­ing down crim­i­nals, there­fore it is un­re­al­is­tic that in the story the peo­ple put so much effort into think­ing what to do with the Baby-eaters. I am not con­vinced. In the real world, you need to al­lo­cate your limited re­sources be­tween many prob­lems you need to deal with. The Baby-eaters are a heretofore un­known prob­lem on a huge scale (pos­si­bly dwarfing all hu­man crim­i­nal­ity), so it makes perfect sense the pro­tag­o­nists would put a lot of effort into deal­ing with it. More­over, we are talk­ing about a fu­ture hu­man­ity in which there is much less vi­o­lent crime (IIRC this is stated ex­plic­itly in the story) and peo­ple are much more sen­si­tive to eth­i­cal is­sues.

  I meant “hu­man uni­ver­sal” not some species-in­de­pen­dent moral­ity.

  I don’t think the story ob­vi­ously pos­tu­lates a hu­man uni­ver­sal moral­ity. It only im­plies that many peo­ple liv­ing at the same time pe­riod have similar views on cer­tain eth­i­cal ques­tions, which doesn’t strike me as un­re­al­is­tic?

  In gen­eral, it feels like this com­ment thread has failed to get to the crux of the dis­agree­ment, and I am not sure if any­thing can be done about it, at least with­out us­ing a more in­ter­ac­tive medium.

  Well, if you feel this is not pro­duc­tive we can stop?

• Vanessa Kosoy 9 Oct 2019 9:30 UTC

  LW: 16 AF: 6

  AF

  in reply to: Lanrian's comment on: Mis­con­cep­tions about con­tin­u­ous takeoff

  Ex­pand­ing on that a lit­tle, even if we know our AIs are mis­al­igned that doesn’t nec­es­sar­ily save us. We might reach a state of knowl­edge when it is easy to cre­ate AIs that (i) mis­al­igned (ii) su­per­hu­man and (iii) non-sin­gu­lar (i.e. a sin­gle such AI is not stronger than the sum to­tal of hu­man­ity and al­igned AIs) but hard/​im­pos­si­ble to cre­ate al­igned su­per­hu­man AIs. Since mis­al­igned AIs that can’t take over still mostly fol­low hu­man in­struc­tions, there will be tremen­dous eco­nomic in­cen­tives to de­ploy more such sys­tems. This is effec­tively a tragedy of the com­mons: for ev­ery in­di­vi­d­ual ac­tor, de­ploy­ing more AIs only in­creases global risk a lit­tle but brings in tremen­dous rev­enue. How­ever, col­lec­tively, risk ac­cu­mu­lates rapidly. At some point the to­tal power of mis­al­igned AIs crosses some (hard to pre­dict in ad­vance) thresh­old and there is a phase tran­si­tion (a cas­cade of failures) from a hu­man-con­trol­led world to a coal­i­tion-of-mis­al­igned-AI-con­trol­led world. Alter­na­tively, the AIs might find a way to ma­nipu­late our en­tire cul­ture into grad­u­ally chang­ing its val­ues into some­thing the AIs pre­fer (like with Mur­der Gandhi).

• Vanessa Kosoy 8 Oct 2019 17:45 UTC

  7 points

  in reply to: shminux's comment on: What do the baby eaters tell us about ethics?

  The “ought”, if any, is purely util­i­tar­ian: to sur­vive, a species has to be slow to act against the morals it finds ab­hor­rent.

  I still don’t un­der­stand, is your claim de­scrip­tive or pre­scrip­tive?

  If you live in a city, there is a 99% chance that there is lit­tle girl within a mile from you be­ing raped and tor­tured by her father/​older brother daily for their own plea­sure, yet no effort is made to find and save her.

  I don’t un­der­stand what you’re say­ing here at all. Ob­vi­ously we have laws against rape, and these laws are en­forced, al­though ofc there are per­pe­tra­tors that don’t get caught. The rea­son these things still hap­pen is clearly not be­cause we, as a species, are tol­er­ant to­wards differ­ent moral­ities.

  My point is that there is no uni­ver­sal we!right and we!wrong in the first place, yet the story was con­structed on this premise, which led to the whole species be­ing hoisted on its own petard.

  “Univer­sal we!right” is a con­tra­dic­tion in terms. The rea­son I added “we” there is be­cause I am talk­ing about our* val­ues, not “uni­ver­sal” val­ues. I agree that there are no uni­ver­sal val­ues. More­over it seems clear to me that, con­trary to your claim, the premise of the story is pre­cisely that there are no uni­ver­sal val­ues. But maybe I just don’t un­der­stand what you’re say­ing here.

  *I was vague about who ex­actly is “we”, but feel free to draw the line around the par­ti­ci­pants of this con­ver­sa­tion any­where you want. Strictly speak­ing, each per­son prob­a­bly has some­what differ­ent val­ues, but in a given de­bate about ethics there might be hope that the par­ti­ci­pants can come to a con­sen­sus. In­deed, if we did not be­lieve there was such hope in this con­ver­sa­tion then there would be no point hav­ing it (at least, to the ex­tent the de­bate is pre­scrip­tive rather than de­scrip­tive, which I am not sure about atm).

• Vanessa Kosoy 8 Oct 2019 11:44 UTC

  4 points

  in reply to: mingyuan's comment on: Who lacks the qualia of con­scious­ness?

  I have some very similar ex­pe­riences*. Dur­ing the “vivid sen­sa­tion” mo­ments (that are un­pleas­ant) I acutely feel that “I am me”, com­pared to which, the rest of the time “I” is just one of the char­ac­ters in the play. In­ci­den­tally, I think that I didn’t have a “vivid sen­sa­tion” mo­ment for years.

  *Or, maybe I don’t have ex­pe­riences since I’m not con­scious? ;)

• Vanessa Kosoy 8 Oct 2019 11:31 UTC

  5 points

  in reply to: shminux's comment on: What do the baby eaters tell us about ethics?

  In what sense is this po­si­tion “un­ten­able”? In the evolu­tion­ary sense (any­one who holds this po­si­tion would go ex­tinct) or in the log­i­cal sense (the po­si­tion can be proved wrong)?

  In evolu­tion, there are two com­pet­ing forces: on the one hand cre­at­ing more con­flict in­creases the prob­a­bil­ity to be de­stroyed, on the other hand im­pos­ing your norms on other groups ob­vi­ously causes those norms to prop­a­gate more and thus makes them more evolu­tion­ary fit. For real hu­mans in the real world, our his­tory kind of is con­stant in­ter­nal in­fight­ing.

  If in­ter­preted in the log­i­cal sense, I don’t think your ar­gu­ment makes sense: it seems like try­ing to de­rive an “ought” from an “is”.

  Also, the ac­tual dis­tance be­tween those di­verg­ing morals mat­ters, and baby eat­ing surely seems like an ex­treme ex­am­ple.

  I don’t claim claim that leav­ing the Baby-eaters alone is nec­es­sar­ily we!wrong, but it is not ob­vi­ous to me that it is we!right, and it is even less ob­vi­ous that it doesn’t make sense to pos­tu­late a fu­ture hu­man cul­ture that would con­sider it right (es­pe­cially that we already know it is sup­posed to be a “weird” cul­ture by mod­ern stan­dards), much less an alien cul­ture like the Su­per-Hap­pies.

Biorisks and X-Risks

• Vanessa Kosoy 7 Oct 2019 12:12 UTC

  7 points

  in reply to: shminux's comment on: What do the baby eaters tell us about ethics?

  I don’t un­der­stand what you’re say­ing here. Do you think it’s im­pos­si­ble for aliens to ex­ist that would want to im­pose their norms on us? Or, you’re say­ing that it was hu­man!wrong of the hu­mans in the story to con­sider med­dling with the baby eaters? What po­si­tion do you think Eliezer en­dorses that you dis­agree with?

• Vanessa Kosoy 6 Oct 2019 22:00 UTC

  2 points

  on: Who lacks the qualia of con­scious­ness?

  A lot of peo­ple seem to as­so­ci­ate con­scious­ness with moral worth. Hence, it seems like ad­mit­ting you lack con­scious­ness is rather dan­ger­ous: you may lose value in the eyes of other peo­ple.

• Vanessa Kosoy 1 Oct 2019 20:52 UTC

  3 points

  in reply to: Raemon's comment on: Notic­ing Frame Differences

  You’re right, if there are more than 2 play­ers, there can be threats and co­op­er­a­tion. How­ever, a lot of the time when game the­o­rists talk about zero-sum games, they talk speci­fi­cally about two-player zero-sum games (be­cause, these games have spe­cial prop­er­ties e.g. the min­i­max the­o­rem).

• Vanessa Kosoy 1 Oct 2019 9:37 UTC

  10 points

  in reply to: Raemon's comment on: Notic­ing Frame Differences

  I un­der­stood what you meant. My point is the the phrase “zero-sum” has very pre­cise mean­ing in game the­ory, and your ex­am­ple doesn’t fit that mean­ing. “Zero-or-nega­tive-sum” is very much not the same as “zero-sum”. In par­tic­u­lar, you called this frame “dom­i­nance and threat”, but in true zero-sum games there are no threats: if you have a way to hurt your op­po­nent, you just use it.

• Vanessa Kosoy 30 Sep 2019 11:42 UTC

  10 points

  on: Notic­ing Frame Differences

  I re­ally liked this post, but I can’t help do­ing a nerdy nit­pick:

  A po­ten­tial differ­ence is that Erica and Frank might think of their situ­a­tion as zero-sum, and there­fore most of the re­s­olu­tion has more to do with figur­ing out “who would win in a poli­ti­cal fight?”, and then hav­ing the coun­ter­fac­tual loser back down.

  The Erica/​Frank game is not zero-sum. The en­tire point is that en­gag­ing in an ac­tual fight would be lower sum than ne­go­ti­at­ing a solu­tion “peace­fully”. From the per­spec­tive of game the­ory, this is a differ­ent type of game with very differ­ent prop­er­ties.

• Vanessa Kosoy 28 Sep 2019 11:28 UTC

  LW: 4 AF: 2

  AF

  in reply to: abramdemski's comment on: For­mal­is­ing de­ci­sion the­ory is hard

  ...But if you model the de­ci­sion as be­ing made over an amount of time, then you could pos­si­bly make the de­ci­sion early on in your de­liber­a­tion, so that the pre­dic­tor can see.

  Yes, sounds sort of rea­son­able. Here is how I think you can re­al­ize this us­ing TRL.

  As usual, we con­sider the agent play­ing an IPD against a pre­dic­tor (New­comb’s para­dox is es­sen­tially play­ing the Pri­soner’s Dilemma against a “FairBot”). On each round, the pre­dic­tor gets to see the agent’s state at the start of the round. (The state can be con­sid­ered part of the “source code”. For ran­dom­iz­ing agents, we also as­sume the pre­dic­tors sees the ran­dom bits). The pre­dic­tor then tries to simu­late the agent (we as­sume it knows the rest of the agent’s source code as well), and is suc­cess­ful if the agent doesn’t ex­e­cute any pro­grams that are too ex­pen­sive for the pre­dic­tor (for the sake of sim­plic­ity, as­sume that no pro­gram started on one round con­tinues run­ning dur­ing fol­low­ing rounds: I don’t think that this as­sump­tion makes a differ­ence of prin­ci­ple). Other­wise, the pre­dic­tion might be wrong (for ex­am­ple, we can as­sume it de­faults to D). The pre­dic­tor then plays D or C ac­cord­ing to its pre­dic­tion of the agent’s ac­tion.

  In this set­ting, the agent can learn the in­com­plete hy­poth­e­sis “if I don’t run ex­pen­sive pro­grams and I play C, the pre­dic­tor will also play C”. (We as­sume that the prior al­lows for side effects of ex­e­cut­ing pro­grams. Such a prior seems more re­al­is­tic any­way, and in par­tic­u­lar is re­quired to counter non-Carte­sian dae­mons. How­ever, it also has a cost, so per­haps what we re­ally want is a prior that is bi­ased to­wards few side effects: but, this is im­ma­te­rial for the cur­rent dis­cus­sion.) This hy­pothe­ses guaran­tees a pay­off of U(CC). As­sum­ing that the pre­dic­tor can­not be ex­ploited, this is the best pos­si­ble pay­off and there­fore the agent will con­verge to co­op­er­a­tion.

  We might want a more ex­plicit re­al­iza­tion of the “simu­lates” part in “agent simu­lates pre­dic­tor”. For this, we can as­sume the agent also re­ceives its own state as an ob­ser­va­tion (but, I’m not sure how gen­er­ally use­ful is this for re­al­is­tic agents). The agent can then also learn the in­com­plete hy­poth­e­sis de­scribing the ex­act func­tion from agent states to pre­dic­tor out­put. How­ever, this hy­poth­e­sis doesn’t af­fect the out­come: it doesn’t pre­dict the agent’s state and there­fore can only guaran­tee the pay­off U(DD).

• Vanessa Kosoy 24 Sep 2019 17:46 UTC

  LW: 4 AF: 2

  AF

  in reply to: abramdemski's comment on: Do Suffi­ciently Ad­vanced Agents Use Logic?

  I had in mind the set­ting where no par­tic­u­lar as­sump­tion is made about the ob­serv­abil­ity of util­ity at all, like in clas­sic set­tings such as Sav­age etc. I’m not claiming it is very rele­vant to the dis­cus­sion at hand. It’s just that there is some no­tion of ra­tio­nal­ity to be stud­ied in such cases. I also mis­spoke in “dire situ­a­tion for learn­ing the­ory”—I don’t want to pre­ma­turely nar­row what learn­ing the­ory can fruit­fully dis­cuss. Some­thing more like “dire situ­a­tion for re­in­force­ment learn­ing” would have been more ac­cu­rate.

  By the way, from the per­spec­tive of what I call “sub­jec­tive” learn­abil­ity, we can con­sider this case as well. In sub­jec­tive learn­abil­ity, we define re­gret with re­spect to an op­ti­mal agent that has all of the user’s knowl­edge (rather than know­ing the ac­tual “true” en­vi­ron­ment). So, it seems not difficult to imag­ine com­mu­ni­ca­tion pro­to­cols that will al­low the AI learn the user’s knowl­edge in­clud­ing the un­ob­serv­able re­ward func­tion (we as­sume the user imag­ines the en­vi­ron­ment as a POMDP with re­wards that are not di­rectly ob­serv­able but defined as a func­tion of the state). (In­ci­den­tally, re­cently I am lean­ing to­wards the po­si­tion that some of kind of generic com­mu­ni­ca­tion pro­to­col (with nat­u­ral lan­guage “an­no­ta­tion” + quan­tiliza­tion to avoid detri­men­tal mes­sages) is the best ap­proach to achieve al­ign­ment.)

• Vanessa Kosoy 24 Sep 2019 16:50 UTC

  LW: 2 AF: 1

  AF

  in reply to: abramdemski's comment on: For­mal­is­ing de­ci­sion the­ory is hard

  Yes, but in this case the agent should two-box.

  Ok, sort of, but then this makes the dis­count­ing “wrong” in the same way that hy­per­bolic dis­count­ing is “wrong”: dy­namic in­con­sis­tency… I wish I could get you to see the other pos­si­ble per­spec­tive, in which there is some­thing go­ing wrong which is not about dis­count­ing be­hav­ior… The two-boxer we are dis­cussing can imag­ine the one-box­ing strat­egy and can have re­gret with re­spect to it.

  Look, a one-box­ing agent would get less util­ity. Yes, it would have higher re­wards on sub­se­quent rounds, but the first round is more im­por­tant, by the very as­sump­tion you made (steep time dis­count). More­over, the two-box­ing agent would get higher util­ity than any agent that one-boxes on some of the rounds, so on all rounds the cor­rect ac­tion is two-box.

  Imag­ine that we let both the ASP pre­dic­tor and the agent think for a while be­fore the first round—so the agent still has a lot more pro­cess­ing power than the pre­dic­tor, but, the pre­dic­tor is likely to get things right even on the first round… The pre­dic­tor can do well w/​o any ac­tual rounds be­ing run be­cause it is a log­i­cal in­duc­tor, so, is learn­ing from closely analo­gous cases across log­i­cal pos­si­bil­ity. Of course for this to make sense I have to as­sume the pre­dic­tor has ac­cess to the agent’s source code. I’m also imag­in­ing that the agent has the pre­dic­tor’s source code.

  I don’t think it makes sense to si­mul­ta­neously as­sume that the agent has a lot more power than the pre­dic­tor and that the pre­dic­tor is likely to get things right even on the first round. If the agent has a lot more power than the pre­dic­tor, then the agent can e.g. di­ag­o­nal­ize against the pre­dic­tor and make sure it will not get things right.

  Once again, we need to ask how does the agent know the pre­dic­tor gets things right on the first round. It needs to learn it some­how. For ex­am­ple, maybe in en­coun­ters many pre­dic­tors one af­ter the other. But then, it will again learn a model in which one-box­ing causes the pre­dic­tor to co­op­er­ate.

  (No­tice that hav­ing a perfect pre­dic­tor re­quires that ei­ther (i) the agent is de­ter­minis­tic or (ii) the pre­dic­tor has ac­cess to the agent’s ran­dom bits. Op­tion (i) is pos­si­ble for the case of com­plete mod­els (be­cause there is a de­ter­minis­tic Bayes-op­ti­mal policy; to es­tab­lish weak fea­si­bil­ity you can use PSRL + pseu­do­ran­dom), which is suffi­cient for New­comb’s para­dox. For in­com­plete mod­els you usu­ally want to ran­dom­ize, al­though it also pos­si­ble to in­stead do de­ter­minis­tic max­imin and op­tion­ally add an ex­ter­nal ran­dom bit gen­er­a­tor which is not a pri­ori as­sumed to be in­visi­ble to the en­vi­ron­ment. In op­tion (ii), you can con­sider a var­i­ant of New­comb’s where the pre­dic­tor re­sponds to the prob­a­bil­ity of the agent mak­ing a par­tic­u­lar choice. This var­i­ant is more difficult for rea­sons similar to what hap­pens with coun­ter­fac­tual mug­ging: it is hard for the agent to dis­cover this be­hav­ior em­piri­cally. I think that, as op­posed to coun­ter­fac­tual mug­ging, you can cap­ture it by an in­com­plete model, al­though it will be a rel­a­tively com­plex, difficult to learn, model.)

  So in some sense you’re think­ing in the “there are in­cor­rect dis­count­ing func­tions” way (ie, steeper is always worse than less steep).

  Well, the agents I con­sider in­ter­est­ing start from a state of ig­no­rance, so they have to learn. For this, the time dis­count has to be shal­low, oth­er­wise there is no time to learn.

• Vanessa Kosoy 21 Sep 2019 13:26 UTC

  LW: 2 AF: 1

  AF

  in reply to: Vanessa Kosoy's comment on: Do Suffi­ciently Ad­vanced Agents Use Logic?

  ...”If agents know a good amount about each other due to ac­cess to source code or other such means, then a sin­gle-shot game has the char­ac­ter of iter­ated game the­ory. How can we cap­ture that in our ra­tio­nal­ity crite­ria?”

  I would study this prob­lem by con­sid­er­ing a pop­u­la­tion of learn­ing agents that are se­quen­tially paired up for one-shot games where the source code of each is re­vealed to the other. This way they can learn how to rea­son about source codes. In con­trast, the model where the agents try to for­mally prove the­o­rems seems poorly mo­ti­vated to me.

  Ac­tu­ally, here’s an­other thought about this. Con­sider the fol­low­ing game: each player sub­mits a pro­gram, then the pro­grams are given each other as in­puts, and ex­e­cuted in any­time mode (i.e. at given mo­ment each pro­gram has to have some an­swer ready). The ex­e­cu­tion has some prob­a­bil­ity $ϵ$ to ter­mi­nate on each time step, so that the to­tal ex­e­cu­tion time fol­lows the ge­o­met­ric dis­tri­bu­tion. Once ex­e­cu­tion is finished, the out­put of the pro­grams is in­ter­preted as strate­gies in a nor­mal-form game and the pay­offs are ac­cord­ingly.

  This seems quite similar to an iter­ated game with ge­o­met­ric time dis­count! In par­tic­u­lar, in this ver­sion of the Pri­soner’s Dilemma, there is the fol­low­ing analogue of the tit-for-tat strat­egy: start by set­ting the an­swer to C, simu­late the other agent, once the other agents starts pro­duc­ing an­swers, set your an­swer to equal to the other agent’s an­swer. For suffi­ciently shal­low time dis­count, this is a Nash equil­ibrium.

  In line with the idea I ex­plained in a pre­vi­ous com­ment, it seems tempt­ing to look for proper/​ther­mal equil­ibria in this game with some con­straint on the pro­grams. One con­straint that seems ap­peal­ing is: force the pro­gram to have $O\left(1\right)$ space com­plex­ity mod­ulo or­a­cle ac­cess to the other pro­gram. It is easy to see that a pair of such pro­grams can be ex­e­cuted us­ing $O\left(1\right)$ space com­plex­ity as well.