Michaël Trazzi

• Michaël Trazzi 12 Jun 2019 17:54 UTC

  3 points

  AF

  in reply to: RyanCarey's comment on: Prob­lems with Coun­ter­fac­tual Oracles

  The string is read with prob­a­bil­ity 1-$ϵ$

• Michaël Trazzi 12 Jun 2019 15:13 UTC

  −1 points

  AF

  in reply to: RyanCarey's comment on: Prob­lems with Coun­ter­fac­tual Oracles

  Yes, if we choose the util­ity func­tion to make it a CDT agent op­ti­miz­ing for the re­ward for one step (so par­tic­u­lar case of act-based) then it won’t care about fu­ture ver­sions of it­self nor want to es­cape.

  I agree with the in­tu­ition of shut­ting down to make it epi­sodic, but I am still con­fused about the causal re­la­tion­ship be­tween “hav­ing the rule to shut­down the sys­tem” and “hav­ing a cur­rent timestep max­i­mizer”. For it to re­ally be a “cur­rent timestep max­i­mizer” it needs to be in some kind of re­ward/​util­ity func­tion. Be­cause ev­ery­thing is re­set at each timestep, there is no in­for­ma­tion point­ing at “I might get shut­down at the next timestep”.

  As for the col­lect­ing a dataset and then op­ti­miz­ing for some nat­u­ral di­rect effect, I am not fa­mil­iar enough with Pearl’s work to tell if that would work, but I made some re­lated com­ments about why there might be some prob­lems in on­line-learn­ing/​”train­ing then test­ing” here.

• Michaël Trazzi 12 Jun 2019 14:53 UTC

  0 points

  AF

  in reply to: Joe_Collman's comment on: Prob­lems with Coun­ter­fac­tual Oracles

  The Asymp­tot­i­cally Unam­bi­tious AGI thread helped me clar­ify my thoughts, thanks. I agree that an op­ti­mal CDT agent won’t think about fu­ture ver­sions, and I don’t see any op­ti­miza­tion pres­sure to­wards es­cape mes­sage nor dis­pro­por­tionately com­mon “es­cape mes­sage” re­gions.

  How­ever, it still as­sumes we have ac­cess to this magic or­a­cle that op­ti­mizes for $R^{\prime }=R.I_E$ where $E$ is the event where hu­mans don’t see the an­swer, $I_E$ its in­di­ca­tor func­tion, and $R$ the coun­ter­fac­tual re­ward (given by the au­to­matic ma­chine). If hu­mans were able to build an or­a­cle perform­ing op­ti­mally (w.r.t $R^{\prime }$) from day 1, then hu­mans would be able to spec­ify some kind of “god or­a­cle”. The rest of the de­sign seems to be just “how to in­ter­act with a god or­a­cle so that hu­mans are not in­fluenced by the an­swers”.

  In prac­tice, you’ll want some­thing that is able to learn from its (ques­tion, pre­dic­tion, re­ward) his­tory. That’s why there is this au­to­matic ma­chine re­ward­ing the or­a­cle with some prob­a­bil­ity $ϵ$. In an on­line learn­ing set­ting, most of the time the model gets $r=0$ (prob­a­bil­ity $1-ϵ$), and it some­times gets some use­ful feed­back $r>0$ (prob­a­bil­ity $ϵ)$. There­fore, if $ϵ$ is small enough (com­pa­rable to the prob­a­bil­ity of hit­ting an es­cape mes­sage at ran­dom), then the learn­ing gets ex­tremely slow and the or­a­cle might shoot at the es­cape ac­tion at ran­dom. Even worse, the (ques­tion, pre­dic­tion, $r=0$) dat­a­points in the his­tory could get so mis­lead­ing that the or­a­cle never learns any­thing.

  Now, if we’re not in an on­line learn­ing pro­cess but in­stead there is a sep­a­ra­tion be­tween a “train­ing phase” and a “de­ploy­ment phase where the AI con­tinue to learns with prob­a­bil­ity $ϵ$”, then the setup is just “have a model that learns to do the use­ful stuff in sand­box, and then have the weights (al­most) fixed in de­ploy­ment”

  In short, I think the CDT setup with­out ma­chine learn­ing as­sumes the prob­lem already solved, that on­line learn­ing won’t work and is un­safe, which leaves us with a “train­ing then de­ploy­ment” setup that isn’t re­ally origi­nal.

• Michaël Trazzi 11 Jun 2019 19:43 UTC

  11 points

  in reply to: Alexei's comment on: Prob­lems with Coun­ter­fac­tual Oracles

  Yes, they call it a low-band­width or­a­cle.

Prob­lems with Coun­ter­fac­tual Oracles

• Michaël Trazzi 3 Jun 2019 14:01 UTC

  6 points

  in reply to: rohinmshah's comment on: Sto­ries of Con­tin­u­ous Deception

  I agree that these sto­ries won’t (nat­u­rally) lead to a treach­er­ous turn. Con­tin­u­ously learn­ing to de­ceive (a ML failure in this case, as you men­tioned) is a differ­ent re­sult. The story/​learn­ing should be sub­stan­tially differ­ent to lead to “learn­ing the con­cept of de­cep­tion” (for reach­ing an AGI-level abil­ity to rea­son about such ab­stract con­cepts), but maybe there’s a way to learn those con­cepts with only nar­row AI.

Sto­ries of Con­tin­u­ous Deception

• Michaël Trazzi 24 May 2019 15:25 UTC

  4 points

  in reply to: Donald Hobson's comment on: Trade-off in AI Ca­pa­bil­ity Concealment

  I in­cluded dates such as 2020 to 2045 to make it more con­crete. I agree that weeks (in­stead of years) would give a more ac­cu­rate rep­re­sen­ta­tion as cur­rent ML ex­per­i­ments take a few weeks tops.

  The sce­nario I had in mind is “in the con­text of a few weeks ML ex­per­i­ment, I achieved hu­man in­tel­li­gence and re­al­ized that I need to con­ceal my in­ten­tions/​ca­pa­bil­ities and I still don’t have de­ci­sive strate­gic ad­van­tage”. The challenge would then be “how to con­ceal my hu­man level in­tel­li­gence be­fore ev­ery­thing I have dis­cov­ered is thrown away”. One way to do this would be to es­cape, for in­stance by copy-past­ing and run­ning your code some­where else.

  If we’re already at the stage of emer­gent hu­man-level in­tel­li­gence from run­ning ML ex­per­i­ments, I would ex­pect “es­cape” to be harder than just hu­man-level in­tel­li­gence (as there would be more con­cerns w.r.t. AGI Safety, and more AI box­ing/​se­cu­rity/​in­ter­pretabil­ity mea­sure), which would ne­ces­sit more re­cur­sive self-im­prove­ment steps, hence more weeks.

  Beside, in such a sce­nario the AI would be in­cen­tivized to spend as much time as pos­si­ble to max­i­mize its true ca­pa­bil­ity, be­cause it would want to max­i­mize its prob­a­bil­ity of suc­cess­fully tak­ing over (be­cause any ex­tra % of tak­ing over would give as­tro­nom­i­cal re­turns in ex­pected value com­pared to just be­ing shut­down).

Trade-off in AI Ca­pa­bil­ity Concealment

• Michaël Trazzi 22 May 2019 10:24 UTC

  6 points

  in reply to: Dagon's comment on: A Treach­er­ous Turn Timeline—Chil­dren, Seed AIs and Pre­dict­ing AI

  Your com­ment makes a lot os sense, thanks.

  I put step 2. be­fore step 3. be­cause I thought some­thing like “first you learn that there is some su­per­vi­sor watch­ing, and then you re­al­ize that you would pre­fer him not to watch”. Agreed that step 2. could hap­pen only by think­ing.

  Yep, de­cep­tion is about al­ign­ment, and I think that most par­ents would be more con­cerned about al­ign­ment, not im­prov­ing the tac­tics. How­ever, I agree that if we take “ed­u­ca­tion” in a broad sense (in­clud­ing high school, col­lege, etc.), it’s un­offi­cially about tac­tics.

  It’s in­ter­est­ing to think of it in terms of co­op­er­a­tion—en­tities less pow­er­ful than their su­per­vi­sors are (in­stru­men­tally) in­cen­tivized to co­op­er­ate.

  what to do with a seed AI that lies, but not so well as to be unnoticeable

  Well, de­stroy it, right? If it’s de­liber­ately do­ing a. or b. (from “Seed AI”) then step 4. has started. The other cases where it could be “ly­ing” from say­ing wrong things would be if its model is con­sis­tently wrong (e.g. stuck in a lo­cal min­ima), so you bet­ter start again from scratch.

  If the su­per­vi­sor isn’t it­self perfectly con­sis­tent and al­igned, some amount of self-de­cep­tion is pre­sent. Any com­pe­tent seed AI (or child) is go­ing to have to learn deception

  That’s in­sight­ful. Bi­ased hu­mans will keep say­ing that they want X when they want Y in­stead, so de­ceiv­ing hu­mans by pre­tend­ing to be work­ing on X while do­ing Y seems in­deed nat­u­ral (as­sum­ing you have “max­i­mize what hu­mans re­ally want” in your code).

• Michaël Trazzi 22 May 2019 9:52 UTC

  4 points

  in reply to: countingtoten's comment on: A Treach­er­ous Turn Timeline—Chil­dren, Seed AIs and Pre­dict­ing AI

  I meant:

  “In my opinion, the dis­agree­ment be­tween Bostrom (treach­er­ous turn) and Go­ertzel (sor­did stum­ble) origi­nates from the un­cer­tainty about how long steps 2. and 3. will take”

  That’s an in­ter­est­ing sce­nario. In­stead of “won’t see a prac­ti­cal way to re­place hu­man­ity with its tools”, I would say “would es­ti­mate its chances of suc­cess to be < 99%”. I agree that we could say that it’s “hon­estly” mak­ing hu­mans happy in the sense that it un­der­stands that this max­i­mizes ex­pected value. How­ever, he knows that there could be much more ex­pected value af­ter re­plac­ing hu­man­ity with its tools, so by do­ing the right thing it’s still “pre­tend­ing” to not know where the ab­surd amount of value is. But yeah, a smile max­i­mizer mak­ing ev­ery­one happy shouldn’t be too con­cerned about con­ceal­ing its ca­pa­bil­ities, short­en­ing step 4.

A Treach­er­ous Turn Timeline—Chil­dren, Seed AIs and Pre­dict­ing AI

• Michaël Trazzi 25 Apr 2019 15:35 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “How use­ful is quan­tiliza­tion for miti­gat­ing speci­fi­ca­tion-gam­ing? (Ryan Carey, Apr. 2019, SafeML ICLR 2019 Work­shop)”

• Michaël Trazzi 25 Apr 2019 15:35 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “Quan­tiliz­ers (Michaël Trazzi & Ryan Carey, Apr. 2019, Github)”.

• Michaël Trazzi 25 Apr 2019 15:35 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “When to use quan­tiliza­tion (Ryan Carey, Feb. 2019, LessWrong)”

• Michaël Trazzi 25 Apr 2019 15:34 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “Quan­tilal con­trol for finite MDPs & Com­put­ing an ex­act quan­tilal policy (Vanessa Kosoy, Apr. 2018, LessWrong)”

• Michaël Trazzi 25 Apr 2019 15:34 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “Re­in­force­ment Learn­ing with a Cor­rupted Re­ward Chan­nel (Tom Ever­itt; Vic­to­ria Krakovna; Lau­rent Orseau; Mar­cus Hut­ter; Shane Legg, Aug. 2017, arXiv; IJCAI)”

• Michaël Trazzi 25 Apr 2019 15:33 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “Thoughts on Quan­tiliz­ers (Stu­art Arm­strong, Jan. 2017, In­tel­li­gent Agent)”

• Michaël Trazzi 25 Apr 2019 15:33 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “Another view of quan­tiliz­ers: avoid­ing Good­hart’s Law (Jes­sica Tay­lor, Jan. 2016, In­tel­li­gent Agent Foun­da­tions Fo­rum)”

• Michaël Trazzi 25 Apr 2019 15:32 UTC

  1 point

  on: Notes on Quantilization

  This thread is to dis­cuss “New pa­per: “Quan­tiliz­ers” (Rob Bens­inger, Nov. 2015, MIRI)”