Michaël Trazzi

• Michaël Trazzi 24 Apr 2019 14:23 UTC

  1 point

  in reply to: Henrik Åslund's comment on: Cor­rigi­bil­ity as Con­strained Optimisation

  Re­ply: The but­ton is a com­mu­ni­ca­tion link be­tween the op­er­a­tor and the agent. In gen­eral, it is pos­si­ble to con­struct an agent that shuts down even though it has re­ceived no such mes­sage from its op­er­a­tors as well as an agent that does get a shut­down mes­sage, but does not shut down. Shut­down is a state de­pen­dent on ac­tions, and not a com­mu­ni­ca­tion link

  This is very clear. Com­mu­ni­ca­tion link made me un­der­stand that it didn’t have a di­rect phys­i­cal effect on the agent. It you want to make it even more in­tu­itive you could do a di­a­gram, but this ex­pla­na­tion is already great!

  Thanks for up­dat­ing the rest of the post and try­ing to make it more clear!

• Michaël Trazzi 11 Apr 2019 11:54 UTC

  1 point

  on: Cor­rigi­bil­ity as Con­strained Optimisation

  Lay­man ques­tions:

  1. I don’t un­der­stand what you mean by “state” in “Sup­pose, how­ever, that the AI lacked any ca­pac­ity to press its shut­down but­ton, or to in­di­rectly con­trol its state”. Do you in­clude its util­ity func­tion in its state? Or just the ob­ser­va­tions he re­ceives from the en­vi­ron­ment? What con­text/​frame­work are you us­ing?

  2. Could you define U_S and U_N? From the Cor­ri­bil­ity pa­per, U_S ap­pears to be an util­ity func­tion fa­vor­ing shut­down, and U_N is a po­ten­tially flawed util­ity func­tion, a first stab at spec­i­fy­ing their own goals. Was that what you meant? I think it’s use­ful to define it in the in­tro­duc­tion.

  3. I don’t un­der­stand how an agent that “[lacks] any ca­pac­ity to press its shut­down but­ton” could have any shut­down abil­ity. It’s seems like a con­tra­dic­tion, un­less you mean “any ca­pac­ity to di­rectly press its shut­down but­ton”.

  4. What’s the “de­fault value func­tion” and the “nor­mal util­ity func­tion” in “Op­ti­mi­sa­tion in­cen­tive”? Is it clearly defined in the lit­ter­a­ture?

  5. “Worse still… for any ac­tion...” → if you choose b as some ac­tion with bad cor­rigi­bil­ity prop­erty, it seems rea­son­able that it can be bet­ter than most ac­tions on v_N + v_S (for in­stance if b is the argmax). I don’t see how that’s a “worse still” sce­nario, it seems plau­si­ble and nor­mal.

  6. “From this rea­son­ing, we con­clude” → are you in­fer­ing things from some hy­po­thetic b that would satisfy all the things you men­tion? If that’s the case, I would need an ex­am­ple to see that it’s in­deed pos­si­ble. Even bet­ter would be a proof that you can always find such b.

  7. “it is clear that we could in the­ory find a θ” → could you ex­pand on this?

  8. “Given the ro­bust op­ti­mi­sa­tion in­cen­tive prop­erty, it is clear that the agent may score very poorly on UN in cer­tain en­vi­ron­ments.” → again, can you ex­pand on why it’s clear?

  9. In the ap­pendix, in your 4 lines in­equal­ity, do you as­sume that U_N(a_s) is non-nega­tive (from line 2 to 3)? If yes, why?

Con­sid­er­ate­ness in OpenAI LP Debate

• Michaël Trazzi 9 Mar 2019 9:26 UTC

  5 points

  on: Re­nam­ing “Front­page”

  Name sug­ges­tions: “ap­proved”, “fa­vored”, “Moder­a­tors’ pick”, “high [in­for­ma­tion] en­tropy”, “origi­nal ideas”, “in­for­ma­tive”, “mostly ideas”.

  More gen­er­ally, I’d recom­mend that each cat­e­gory has a name that bluntly states what the filter does (e.g. if it only uses karma as filter say “high karma”).

• Michaël Trazzi 8 Mar 2019 21:57 UTC

  43 points

  AF

  on: Align­ment Re­search Field Guide

  Hey Abram (and the MIRI re­search team)!

  This post res­onates with me on so many lev­els. I vividly re­mem­ber the Hu­man-Aligned AI Sum­mer School where you used to be a “re­ceiver” and Vlad was a “trans­mit­ter”, when talk­ing about “op­ti­miz­ers”. Your “doc­u­ment” es­pe­cially res­onates with my ex­pe­rience run­ning an AI Safety Meetup (Paris AI Safety).

  On Jan­uary 2019, I or­ga­nized a Meetup about “Deep RL from hu­man prefer­ences”. Essen­tially, the re­sources were by difficulty, so you could dis­cuss the 80k pod­cast, the open AI blog­post, the origi­nal pa­per or even a re­cent rele­vant pa­per. Even if the par­ti­ci­pants were “fa­mil­iar” to RL (be­cause they got used to see writ­ten “RL” in blogs or hear peo­ple say “RL” in pod­casts) none of them could ex­plain to me the core struc­ture of a RL set­ting (i.e. that a RL prob­lem would need at least an en­vi­ron­ment, ac­tions, etc.)

  The boys were get­ting hun­gry (abram is right, $10 of chips is not enough for 4 hun­gry men be­tween 7 and 9pm), when in the mid­dle of a monologue (“in RL, you have so-and-so, and then it goes like so on and so forth...“), I sud­denly re­al­ize that I’m talk­ing to more than qual­ified at­ten­dees (I was lucky to have a PhD can­di­date in eco­nomics, a teenager who used to do in­ter­na­tional olympiads in in­for­mat­ics (IOI) and a CS PhD) that lack the nec­es­sary RL pro­ce­du­ral knowl­edge to ask non-triv­ial ques­tions about “Deep RL from hu­man prefer­ences”.

  That’s when I de­cided to change the lo­gis­tics of the Meetup to some­thing much closer to what is de­scribed in “You and your re­search”. I started think­ing about what they would be in­ter­ested in know­ing. So I started tel­ling the brillant IOI kid about this MIRI sum­mer pro­gram, how I ap­plied last year, etc. One thing lead to an­other, and I ended up ask­ing what Tsvi had asked me one year ago for the AISFP in­ter­view:

  If one of you was the only Align­ment re­searcher left on Earth, and it was for­bid­den to con­vince other peo­ple to work on AI Safety re­search, what would you do?

  That got ev­ery­one ex­cited. The IOI boy took the black marker, and started to do math to the ques­tion, as a trans­mit­ter: “So, there is a prob­a­bil­ity p_0 that AI Re­searchers will solve the prob­lem with­out me, and p_1 that my con­tri­bu­tion will be neg-util­ity, so if we as­sume this and that, we get so-and-so.”

  The mo­ment I asked ques­tions I was truly cu­ri­ous about, the Meetup went from a po­lite gath­er­ing to the most in­ter­est­ing dis­cus­sion of 2019.

  Abram, if I were in charge of all agents in the refer­ence class “or­ga­nizer of Align­ment-re­lated events”, I would tell in­stances of that class with my spe­cific char­ac­ter­is­tics two things:

  1. Come back to this doc­u­ment be­fore and af­ter ev­ery Meetup.

  2. Please write be­low (can be in this thread or in the com­ments) what was your ex­pe­rience run­ning an Align­ment think-thank that res­onates the most with the above “doc­u­ment”.

Med­i­ta­tions on Territory

Treach­er­ous Turn, Si­mu­la­tions and Brain-Com­puter Interfaces

• Michaël Trazzi 5 Feb 2019 23:14 UTC

  7 points

  in reply to: rohinmshah's comment on: Great­est Lower Bound for AGI

  I in­tu­itively agree with your an­swer. Av­turchin also com­mented say­ing some­thing close (he said 2019, but for differ­ent rea­sons). There­fore, I think I might not be com­mu­ni­cat­ing clearly my con­fu­sion.

  I don’t re­mem­ber ex­actly when, but there was some de­bates be­tween Yann Le Cun and AI Align­ment folks on a Fb group (maybe AI Safety dis­cus­sion “open” a few months ago). What stroke me was how con­fi­dent LeCun was about long timelines. I think, for him, the 1% would be in at least 10 years. How do you ex­plain that some­one who has ac­cess to pri­vate in­for­ma­tion (e.g. at FAIR) might have timelines so differ­ent than yours?

  Meta: Thanks for ex­press­ing clearly your con­fi­dence lev­els through your writ­ing with “hard”, “maybe” and “should”: it’s very effi­cient.

  EDIT: Le Cun thread: https://​​www.face­book.com/​​groups/​​aisafety/​​perma­l­ink/​​1178285709002208/​​

• Michaël Trazzi 5 Feb 2019 23:06 UTC

  4 points

  in reply to: avturchin's comment on: Great­est Lower Bound for AGI

  Could you de­tail a bit more the Gott’s equa­tion? I’m not fa­mil­iar with it.

  Also, do you think that those 62 years are mean­ingful if we think about AI win­ters or ex­po­nen­tial tech­nolog­i­cal progress?

  PS: I think you com­mented in­stead of giv­ing an an­swer (differ­ent things in ques­tion posts)

[Question] Great­est Lower Bound for AGI

• Michaël Trazzi 23 Nov 2018 23:17 UTC

  5 points

  on: If You Want to Win, Stop Conceding

  Thanks for the post!

  It res­onates with some ex­pe­rience I had in play­ing the game of go at a com­pet­i­tive level.

  Go is a perfect in­for­ma­tion game but it’s very hard to know ex­actly what will be the out­come of a “fight” (you would need to look up to 30 moves ahead in some cases).

  So when the other guy would kill your group of stones af­ter a “life or death” sce­nario, be­cause he had a slight ad­van­tage in the fight, it feels like the other is lucky, and most peo­ple have re­ally bad thoughts and just give up.

  Once, I cre­ated an ac­count with the bio “I don’t re­sign” to see what would hap­pen if I forced my­self to not con­cede and keep play­ing af­ter a big loss. It went sur­pris­ingly well and I even went to play the high­est ranked guy con­nected on the server. At this point, I com­pletely lost the game and there was 100+ peo­ple watch­ing the game, so I just re­signed.

  Look­ing back, it definitely helped me to con­tinue fight­ing even af­ter a big loss, and stop the men­tal chat­ter. How­ever, there’s a trade-off be­tween the time gained by cor­rectly es­ti­mat­ing the prob­a­bil­ity of win­ning and re­sign­ing when too im­prob­a­ble, and the men­tal en­ergy gained from not re­sign­ing (minus the fact that your op­po­nent may be pretty pissed off).

• Michaël Trazzi 31 Oct 2018 11:49 UTC

  3 points

  in reply to: habryka's comment on: In­tro­duc­ing the AI Align­ment Fo­rum (FAQ)

  (the ac­count databases are shared, so ev­ery LW user can log in on al­ign­ment fo­rum, but it will say “not a mem­ber” in the top right cor­ner)

  I am hav­ing some is­sues in try­ing to log in from a github-linked ac­count. It redi­rects me to LW with an empty page and does noth­ing.

• Michaël Trazzi 16 Oct 2018 11:37 UTC

  2 points

  on: notic­ing in­ter­nal experiences

  This web­site is de­signed to make you write about three morn­ing pages ev­ery day.

  I’ve used it for about two years and wrote ~200k words.

  Really recom­mend it to form an habit of daily free writ­ing.

• Michaël Trazzi 14 Oct 2018 20:55 UTC

  2 points

  in reply to: Said Achmiz's comment on: Open Thread Oc­to­ber 2018

  Same is­sue here with the <a class=“users-name” href=“/​users/​mtrazzi”>Michaël Trazzi</​a> tag. The e in “ë” is larger than the “a” (here is a pic­ture).

  The bug seems to come from font-fam­ily: warnock-pro,Palat­ino,“Palat­ino Lino­type”,“Palat­ino LT STD”,“Book An­tiqua”,Ge­or­gia,serif;” in .Post­sPage-au­thor (in <style data-jss=“” data-meta=“Post­sPage”>).

  If I delete this font-fam­ily line, the font changes but the “ë” (and any other let­ter with ac­cent) ap­pears to have the cor­rect size.

Open Thread Oc­to­ber 2018

• Michaël Trazzi 11 Sep 2018 9:11 UTC

  1 point

  in reply to: Said Achmiz's comment on: A Dialogue on Ra­tion­al­ist Activism

  You: Well.

  The “You” should be bold.

• Michaël Trazzi 1 Sep 2018 7:38 UTC

  3 points

  on: For­mal vs. Effec­tive Pre-Commitment

  typo: “Ca­sual De­ci­sion The­ory”

• Michaël Trazzi 31 Aug 2018 20:15 UTC

  8 points

  on: Bot­tle Caps Aren’t Optimisers

  Let me see if I got it right:

  1. Defin­ing op­ti­miz­ers as an un­pre­dictable pro­cess max­i­miz­ing an ob­jec­tive func­tion does not take into ac­count al­gorithms that we can compute

  2. Satis­fy­ing the prop­erty P “give the ob­jec­tive func­tion higher val­ues than an in­ex­is­tence baseline” is not suffi­cient:

  • the lid satis­fies (P) with “wa­ter quan­tity in bot­tle” but is just a rigid ob­ject that some op­ti­mizer put there. How­ever, not the best counter-ex­am­ple be­cause not a Yud­kwoskian op­ti­mizer.

  • if a liver didn’t ex­ist or did other ran­dom things then hu­mans wouldn’t be al­ive and rich, so it satis­fies (P) with “money in bank ac­count” as the ob­jec­tive func­tion. How­ever, the bet­ter way to ac­count for its be­havi­our (cf. Yud­kowskian defi­ni­tion) is to see it as a sub-pro­cess of an in­come max­i­mizer cre­ated by evolu­tion.

  1. One prop­erty that could work: have a step in the al­gorithm that prov­ably aug­ments the ob­jec­tive func­tion (e.g. gra­di­ent as­cent).

  Prop­er­ties I think are rele­vant:

  • in­tent: the lid did not “chose” to be there, hu­mans did

  • do­ing some­thing that the outer op­ti­mizer can­not do “as well” with­out us­ing the same pro­cess as the in­ner op­ti­mizer : would be very tiring for hu­mans to use our hands as lids. Hu­mans can­not play go as well as Alpha Zero with­out ac­tu­ally run­ning the al­gorithm.

• Michaël Trazzi 29 Aug 2018 13:44 UTC

  4 points

  in reply to: rohinmshah's comment on: HLAI 2018 Field Report

  it feels wrong to call other re­search dan­ger­ous, es­pe­cially given its enor­mous po­ten­tial for good.

  I agree that call­ing 99.9% of AI re­search “dan­ger­ous” and AI Safety re­search “safe” is not an use­ful di­chotomy. How­ever, I con­sider AGI com­pa­nies/​labs and peo­ple fo­cus­ing on im­ple­ment­ing self-im­prov­ing AI/​code syn­the­sis ex­tremely dan­ger­ous. Same for any break­through in gen­eral AI, or things that greatly shorten the AGI timeline.

  Do you mean that some AI re­search have pos­i­tive ex­pected util­ity (e.g. in medecine) and should not be called dan­ger­ous be­cause the good they pro­duce com­pen­sates for the in­creased AI-risk?

• Michaël Trazzi 29 Aug 2018 13:06 UTC

  12 points

  on: HLAI 2018 Field Report

  out­side that bub­ble peo­ple still don’t know or have con­fused ideas about how it’s dan­ger­ous, even among the group of peo­ple weird enough to work on AGI in­stead of more aca­dem­i­cally re­spectable, nar­row AI.

  I agree. I run a lo­cal AI Safety Meetup and it’s frus­trat­ing to see that the ones who bet­ter un­der­stand the dis­cussed con­cepts con­sider that Safety is way less in­ter­est­ing/​im­por­tant than AGI Ca­pa­bil­ities re­search. I re­mem­ber some­one say­ing some­thing like: “Ok, this Safety thing is kind of in­ter­est­ing, but who would be in­ter­ested in work­ing on real AGI prob­lems” and the other guys nod­ing. What they say:

  • “I’ll start an AGI re­search lab. When I feel we’re close enough to AGI I’ll con­sider Safety.”

  • “It’s difficult to do sig­nifi­cant re­search on Safety with­out know­ing a lot about AI in gen­eral.”