Vika

• Vika 11 May 2019 3:50 UTC

  6 points

  AF

  in reply to: rohinmshah's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  Thanks Ro­hin! Your ex­pla­na­tions (both in the com­ments and offline) were very helpful and clar­ified a lot of things for me. My cur­rent un­der­stand­ing as a re­sult of our dis­cus­sion is as fol­lows.

  AU is a func­tion of the world state, but in­tends to cap­ture some gen­eral mea­sure of the agent’s in­fluence over the en­vi­ron­ment that does not de­pend on the state rep­re­sen­ta­tion.

  Here is a hi­er­ar­chy of ob­jects, where each ob­ject is a func­tion of the pre­vi­ous one: world states /​ microstates (e.g. quark con­figu­ra­tion) → ob­ser­va­tions (e.g. pix­els) → state rep­re­sen­ta­tion /​ coarse-grain­ing (which defines macrostates as equiv­alence classes over ob­ser­va­tions) → fea­tur­iza­tion (a coarse-grain­ing that fac­tor­izes into fea­tures). The im­pact mea­sure is defined over the macrostates.

  Con­sider the set of all state rep­re­sen­ta­tions that are con­sis­tent with the true re­ward func­tion (i.e. if two microstates have differ­ent true re­wards, then their state rep­re­sen­ta­tion is differ­ent). The im­pact mea­sure is rep­re­sen­ta­tion-in­var­i­ant if it has the same val­ues for any state rep­re­sen­ta­tion in this re­ward-com­pat­i­ble set. (Note that if rep­re­sen­ta­tion in­var­i­ance was defined over the set of all pos­si­ble state rep­re­sen­ta­tions, this set would in­clude the most coarse-grained rep­re­sen­ta­tion with all ob­ser­va­tions in one macrostate, which would im­ply that the im­pact mea­sure is always 0.) Now con­sider the most coarse-grained rep­re­sen­ta­tion R that is con­sis­tent with the true re­ward func­tion.

  An AU mea­sure defined over R would re­main the same for a finer-grained rep­re­sen­ta­tion. For ex­am­ple, if the at­tain­able set con­tains a re­ward func­tion that re­wards hav­ing a vase in the room, and the rep­re­sen­ta­tion is re­fined to dis­t­in­guish green and blue vases, then macrostates with differ­ent-col­ored vases would re­ceive the same re­ward. Thus, this mea­sure would be rep­re­sen­ta­tion-in­var­i­ant. How­ever, for an AU mea­sure defined over a finer-grained rep­re­sen­ta­tion (e.g. dis­t­in­guish­ing blue and green vases), a ran­dom re­ward func­tion in the at­tain­able set could as­sign a differ­ent re­ward to macrostates with blue and green vases, and the re­sult­ing mea­sure would be differ­ent from the mea­sure defined over R.

  An RR mea­sure that only uses reach­a­bil­ity func­tions of sin­gle macrostates is not rep­re­sen­ta­tion-in­var­i­ant, be­cause the ob­ser­va­tions in­cluded in each macrostate de­pend on the coarse-grain­ing. How­ever, if we al­low the RR mea­sure to use reach­a­bil­ity func­tions of sets of macrostates, then it would be rep­re­sen­ta­tion-in­var­i­ant if it is defined over R. Then a func­tion that re­wards reach­ing a macrostate with a vase can be defined in a finer-grained rep­re­sen­ta­tion by re­ward­ing macrostates with green or blue vases. Thus, both AU and this ver­sion of RR are rep­re­sen­ta­tion-in­var­i­ant iff they are defined over the most coarse-grained rep­re­sen­ta­tion con­sis­tent with the true re­ward.

• Vika 3 May 2019 13:44 UTC

  6 points

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  There are var­i­ous parts of your ex­pla­na­tion that I find vague and could use a clar­ifi­ca­tion on:

  • “AUP is not about state”—what does it mean for a method to be “about state”? Same goes for “the di­rect fo­cus should not be on the state”—what does “di­rect fo­cus” mean here?

  • “Overfit­ting the en­vi­ron­ment”—I know what it means to overfit a train­ing set, but I don’t know what it means to overfit an en­vi­ron­ment.

  • “The long arms of op­por­tu­nity cost and in­stru­men­tal con­ver­gence”—what do “long arms” mean?

  • “Wire­head a util­ity func­tion”—is this the same as op­ti­miz­ing a util­ity func­tion?

  • “Cut out the mid­dle­man”—what are you refer­ring to here?

  I think these in­tu­itive phrases may be a use­ful short­hand for some­one who already un­der­stands what you are talk­ing about, but since I do not un­der­stand, I have not found them illu­mi­nat­ing.

  I sym­pa­thize with your frus­tra­tion about the difficulty of com­mu­ni­cat­ing these com­plex ideas clearly. I think the difficulty is caused by the vague lan­guage rather than miss­ing key ideas, and mak­ing the lan­guage more pre­cise would go a long way.

• Vika 2 May 2019 17:01 UTC

  6 points

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  Thanks for the de­tailed ex­pla­na­tion—I feel a bit less con­fused now. I was not in­tend­ing to ex­press con­fi­dence about my pre­dic­tion of what AU does. I was aware that I didn’t un­der­stand the state rep­re­sen­ta­tion in­var­i­ance claim in the AUP pro­posal, though I didn’t re­al­ize that it is as cen­tral to the pro­posal as you de­scribe here.

  I am still con­fused about what you means by pe­nal­iz­ing ‘power’ and what ex­actly it is a func­tion of. The way you de­scribe it here sounds like it’s a mea­sure of the agent’s op­ti­miza­tion abil­ity that does not de­pend on the state at all. Did you mean that in the real world the agent always re­ceives the same AUP penalty no mat­ter which state it is in? If that is what you meant, then I’m not sure how to rec­on­cile your de­scrip­tion of AUP in the real world (where the penalty is not a func­tion of the state) and AUP in an MDP (where it is a func­tion of the state). I would find it helpful to see a defi­ni­tion of AUP in a POMDP as an in­ter­me­di­ate case.

  I agree with Daniel’s com­ment that if AUP is not pe­nal­iz­ing effects on the world, then it is con­fus­ing to call it an ‘im­pact mea­sure’, and some­thing like ‘op­ti­miza­tion reg­u­lariza­tion’ would be bet­ter.

  Since I still have lin­ger­ing con­fu­sions af­ter your lat­est ex­pla­na­tion, I would re­ally ap­pre­ci­ate if some­one else who un­der­stands this could ex­plain it to me.

• Vika 22 Apr 2019 17:36 UTC

  2 points

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  Are you think­ing of an ac­tion ob­ser­va­tion for­mal­ism, or some kind of re­ward func­tion over in­ferred state?

  I don’t quite un­der­stand what you’re ask­ing here, could you clar­ify?

  If you had to pose the prob­lem of im­pact mea­sure­ment as a ques­tion, what would it be?

  Some­thing along the lines of: “How can we mea­sure to what ex­tent the agent is chang­ing the world in ways that we care about?”. Why?

• Vika 20 Apr 2019 13:23 UTC

  2 points

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  What does this mean, con­cretely? And what hap­pens with the sur­vival util­ity func­tion be­ing the sole mem­ber of the at­tain­able set? Does this run into that prob­lem, in your model?

  I meant that for at­tain­able set con­sist­ing of ran­dom util­ity func­tions, I would ex­pect most of the vari­a­tion in util­ity to be based on ir­rele­vant fac­tors like the po­si­tions of air molecules. This does not ap­ply to the at­tain­able set con­sist­ing of the sur­vival util­ity func­tion, since that is not a ran­dom util­ity func­tion.

  What makes you think that?

  This is an in­tu­itive claim based on a gen­eral ob­ser­va­tion of how peo­ple at­tribute re­spon­si­bil­ity. For ex­am­ple, if I walk into a busy street and get hit by a car, I will be con­sid­ered re­spon­si­ble for this be­cause it’s easy to pre­dict. On the other hand, if I am walk­ing down the street and a brick falls on my head from the nearby build­ing, then I will not be con­sid­ered re­spon­si­ble, be­cause this event would be hard to pre­dict. There are prob­a­bly other rea­sons that hu­mans don’t con­sider them­selves re­spon­si­ble for but­terfly effects.

• Vika 19 Apr 2019 12:51 UTC

  14 points

  AF

  on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  Thanks Alex for start­ing this dis­cus­sion and thanks ev­ery­one for the thought-pro­vok­ing an­swers. Here is my cur­rent set of con­cerns about the use­ful­ness of im­pact mea­sures, sorted in de­creas­ing or­der of con­cern:

  Ir­rele­vant fac­tors. When ap­plied to the real world, im­pact mea­sures are likely to be dom­i­nated by things hu­mans don’t care about (heat dis­si­pa­tion, con­vec­tion cur­rents, po­si­tions of air molecules, etc). This seems likely to hap­pen to value-ag­nos­tic im­pact mea­sures, e.g. AU with ran­dom util­ity func­tions, which would mostly end up re­ward­ing spe­cific con­figu­ra­tions of air molecules.

  This may be miti­gated by in­abil­ity to per­ceive the ir­rele­vant fac­tors, which re­sults in a more coarse-grained state rep­re­sen­ta­tion: if the agent can’t see air molecules, all the states with differ­ent air molecule po­si­tions will look the same, as they do to hu­mans. Some hu­man-rele­vant fac­tors can also be difficult to per­ceive, e.g. the pres­ence of poi­sonous gas in the room, so we may not want to limit the agent’s per­cep­tion abil­ity to hu­man level. Au­to­mat­i­cally fil­ter­ing out ir­rele­vant fac­tors does seem difficult, and I think this might im­ply that it is im­pos­si­ble to de­sign an im­pact mea­sure that is both use­ful and truly value-ag­nos­tic.

  How­ever, the value-ag­nos­tic crite­rion does not seem very im­por­tant in it­self. I think the rele­vant crite­rion is that de­sign­ing im­pact mea­sures should be eas­ier than the gen­eral value learn­ing prob­lem. We already have a non-value-ag­nos­tic im­pact mea­sure that plau­si­bly satis­fies this crite­rion: RLSP learns what is effec­tively an im­pact mea­sure (the hu­man theta pa­ram­e­ter) us­ing zero hu­man in­put just by ex­am­in­ing the start­ing state. This could also po­ten­tially be achieved by choos­ing an at­tain­able util­ity set that re­wards a broad enough sam­ple of things hu­mans care about, and leaves the rest to gen­er­al­iza­tion. Choos­ing a good at­tain­able util­ity set may not be easy but it seems un­likely to be as hard as the gen­eral value learn­ing prob­lem.

  But­terfly effects. Every ac­tion is likely to have large effects that are difficult to pre­dict, e.g. tak­ing a differ­ent route to work may re­sult in differ­ent peo­ple be­ing born. Taken liter­ally, this means that there is no such thing as a low-im­pact ac­tion. Hu­mans get around this by only count­ing eas­ily pre­dictable effects as im­pact that they are con­sid­ered re­spon­si­ble for. If we fol­low a similar strat­egy of not pe­nal­iz­ing but­terfly effects, we might in­cen­tivize the agent to de­liber­ately cause but­terfly effects. The eas­iest way around this that I can cur­rently see is re­strict­ing the agent’s ca­pa­bil­ity to model the effects of its ac­tions, though this has ob­vi­ous use­ful­ness costs as well.

  Chaotic world. Every ac­tion, in­clud­ing in­ac­tion, is ir­re­versible, and each branch con­tains differ­ent states. While pre­serv­ing re­versibil­ity is im­pos­si­ble in this world, pre­serv­ing op­tion­al­ity (at­tain­able util­ity, reach­a­bil­ity, etc) seems pos­si­ble. For ex­am­ple, if the at­tain­able set con­tains a func­tion that re­wards the pres­ence of vases, the ac­tion of break­ing a vase will make this re­ward func­tion more difficult to satisfy (even if the states with/​with­out vases are differ­ent in ev­ery branch). If we solve the prob­lem of de­sign­ing/​learn­ing a good util­ity set that is not dom­i­nated by ir­rele­vant fac­tors, I ex­pect chaotic effects will not be an is­sue.

  If any of the above-men­tioned con­cerns are not over­come, im­pact mea­sures will fail to dis­t­in­guish be­tween what hu­mans would con­sider low-im­pact and high-im­pact. Thus, pe­nal­iz­ing high-im­pact ac­tions would come with pe­nal­iz­ing low-im­pact ac­tions as well, which would re­sult in a strong safety-ca­pa­bil­ity trade­off. I think the most in­for­ma­tive di­rec­tion of re­search to figure out whether these con­cerns are a deal-breaker is to scale up im­pact mea­sures to ap­ply be­yond grid­wor­lds, e.g. to Atari games.

• Vika 11 Apr 2019 15:16 UTC

  9 points

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  I don’t see how rep­re­sen­ta­tion in­var­i­ance ad­dresses this con­cern. As far as I un­der­stand, the con­cern is about any ac­tions in the real world caus­ing large but­terfly effects. This in­cludes effects that would be cap­tured by any rea­son­able rep­re­sen­ta­tion, e.g. differ­ent peo­ple ex­ist­ing in the ac­tion and in­ac­tion branches of the world. The state rep­re­sen­ta­tions used by hu­mans also dis­t­in­guish be­tween these world branches, but hu­mans have limited mod­els of the fu­ture that don’t cap­ture but­terfly effects (e.g. per­son X can dis­t­in­guish be­tween the world state where per­son Y ex­ists and the world state where per­son Z ex­ists, but can’t pre­dict that choos­ing a differ­ent route to work will cause per­son Z to ex­ist in­stead of per­son Y).

  I agree with Daniel that this is a ma­jor prob­lem with im­pact mea­sures. I think that to get around this prob­lem we would ei­ther need to figure out how to dis­t­in­guish but­terfly effects from other effects (and then in­clude all the but­terfly effects in the in­ac­tion branch) or use a weak world model that does not cap­ture but­terfly effects (similarly to hu­mans) for mea­sur­ing im­pact. Even if we know how to do this, it’s not en­tirely clear whether we should avoid pe­nal­iz­ing but­terfly effects. Un­like hu­mans, AI sys­tems would be able to cause but­terfly effects on pur­pose, and could chan­nel their im­pact through but­terfly effects if they are not pe­nal­ized.

De­sign­ing agent in­cen­tives to avoid side effects

New safety re­search agenda: scal­able agent al­ign­ment via re­ward modeling

• Vika 10 Nov 2018 18:48 UTC

  4 points

  AF

  in reply to: Samuel Rødal's comment on: Speci­fi­ca­tion gam­ing ex­am­ples in AI

  As a re­sult of the re­cent at­ten­tion, the speci­fi­ca­tion gam­ing list has re­ceived a num­ber of new sub­mis­sions, so this is a good time to check out the lat­est ver­sion :).

• Vika 1 Nov 2018 21:18 UTC

  2 points

  AF

  in reply to: habryka's comment on: Dis­cus­sion on the ma­chine learn­ing ap­proach to AI safety

  Awe­some, thanks Oliver!

Dis­cus­sion on the ma­chine learn­ing ap­proach to AI safety

• Vika 12 Oct 2018 16:01 UTC

  4 points

  AF

  in reply to: TurnTrout's comment on: Towards a New Im­pact Measure

  Thanks, glad you liked the break­down!

  The agent would have an in­cen­tive to stop any­one from do­ing any­thing new in re­sponse to what the agent did

  I think that the step­wise coun­ter­fac­tual is suffi­cient to ad­dress this kind of cling­i­ness: the agent will not have an in­cen­tive to take fur­ther ac­tions to stop hu­mans from do­ing any­thing new in re­sponse to its origi­nal ac­tion, since af­ter the origi­nal ac­tion hap­pens, the hu­man re­ac­tions are part of the step­wise in­ac­tion baseline.

  The penalty for the origi­nal ac­tion will take into ac­count hu­man re­ac­tions in the in­ac­tion rol­lout af­ter this ac­tion, so the agent will pre­fer ac­tions that re­sult in hu­mans chang­ing fewer things in re­sponse. I’m not sure whether to con­sider this cling­i­ness—if so, it might be use­ful to call it “ex ante cling­i­ness” to dis­t­in­guish from “ex post cling­i­ness” (similar to your cor­re­spond­ing dis­tinc­tion for offset­ting). The “ex ante” kind of cling­i­ness is the same prop­erty that causes the agent to avoid scape­goat­ing but­terfly effects, so I think it’s a de­sir­able prop­erty over­all. Do you dis­agree?

New Deep­Mind AI Safety Re­search Blog

• Vika 25 Sep 2018 16:36 UTC

  5 points

  AF

  on: Align­ment Newslet­ter #25

  Thanks Ro­hin for a great sum­mary as always!

  I think the prop­erty of han­dling shut­down de­pends on the choice of ab­solute value or trun­ca­tion at 0 in the de­vi­a­tion mea­sure, not the choice of the core part of the de­vi­a­tion mea­sure. RR doesn’t han­dle shut­down be­cause by de­fault it is set to only pe­nal­ize re­duc­tions in reach­a­bil­ity (us­ing trun­ca­tion at 0). I would ex­pect that re­plac­ing the trun­ca­tion with ab­solute value (thus pe­nal­iz­ing in­creases in reach­a­bil­ity as well) would re­sult in han­dling shut­down (but break the asym­me­try prop­erty from the RR pa­per). Similarly, AUP could be mod­ified to only pe­nal­ize re­duc­tions in goal-achiev­ing abil­ity by re­plac­ing the ab­solute value with trun­ca­tion, which I think would make it satisfy the asym­me­try prop­erty but not han­dle shut­down.

  More thoughts on in­de­pen­dent de­sign choices here.

• Vika 24 Sep 2018 18:39 UTC

  19 points

  AF

  on: Towards a New Im­pact Measure

  There are sev­eral in­de­pen­dent de­sign choices made by AUP, RR, and other im­pact mea­sures, which could po­ten­tially be used in any com­bi­na­tion. Here is a break­down of de­sign choices and what I think they achieve:

  Baseline

  • Start­ing state: used by re­versibil­ity meth­ods. Re­sults in in­terfer­ence with other agents. Avoids ex post offset­ting.

  • In­ac­tion (ini­tial branch): de­fault set­ting in Low Im­pact AI and RR. Avoids in­terfer­ing with other agent’s ac­tions, but in­terferes with their re­ac­tions. Does not avoid ex post offset­ting if the penalty for pre­vent­ing events is nonzero.

  • In­ac­tion (step­wise branch) with en­vi­ron­ment model rol­louts: de­fault set­ting in AUP, model rol­louts are nec­es­sary for pe­nal­iz­ing de­layed effects. Avoids in­terfer­ence with other agents and ex post offset­ting.

  Core part of de­vi­a­tion measure

  • AUP: differ­ence in at­tain­able util­ities be­tween baseline and cur­rent state

  • RR: differ­ence in state reach­a­bil­ity be­tween baseline and cur­rent state

  • Low im­pact AI: dis­tance be­tween baseline and cur­rent state

  Func­tion ap­plied to core part of de­vi­a­tion measure

  • Ab­solute value: de­fault set­ting in AUP and Low Im­pact AI. Re­sults in pe­nal­iz­ing both in­crease and re­duc­tion rel­a­tive to baseline. This re­sults in avoid­ing the sur­vival in­cen­tive (satis­fy­ing the Cor­rigi­bil­ity prop­erty given in AUP post) and in equal penalties for pre­vent­ing and caus­ing the same event (vi­o­lat­ing the Asym­me­try prop­erty given in RR pa­per).

  • Trun­ca­tion at 0: de­fault set­ting in RR, re­sults in pe­nal­iz­ing only re­duc­tion rel­a­tive to baseline. This re­sults in un­equal penalties for pre­vent­ing and caus­ing the same event (satis­fy­ing the Asym­me­try prop­erty) and in not avoid­ing the sur­vival in­cen­tive (vi­o­lat­ing the Cor­rigi­bil­ity prop­erty).

  Scaling

  • Hand-tuned: de­fault set­ting in RR (sort of pro­vi­sion­ally)

  • Im­pactUnit: used by AUP

  I think an ab­la­tion study is needed to try out differ­ent com­bi­na­tions of these de­sign choices and in­ves­ti­gate which of them con­tribute to which desider­ata /​ ex­per­i­men­tal test cases. I in­tend to do this at some point (hope­fully soon).

• Vika 23 Sep 2018 19:52 UTC

  2 points

  AF

  in reply to: Vika's comment on: Towards a New Im­pact Measure

  Another is­sue with equally pe­nal­iz­ing de­creases and in­creases in power (as AUP does) is that for any event A, it equally pe­nal­izes the agent for caus­ing event A and for pre­vent­ing event A (vi­o­lat­ing prop­erty 3 in the RR pa­per). I origi­nally thought that satis­fy­ing Prop­erty 3 is nec­es­sary for avoid­ing ex post offset­ting, which is ac­tu­ally not the case (ex post offset­ting is caused by pe­nal­iz­ing the given ac­tion on fu­ture time steps, which the step­wise in­ac­tion baseline avoids). How­ever, I still think it’s bad for an im­pact mea­sure to not dis­t­in­guish be­tween cau­sa­tion and pre­ven­tion, es­pe­cially for ir­re­versible events.

  This comes up in the car driv­ing ex­am­ple already men­tioned in other com­ments on this post. The rea­son the ac­tion of keep­ing the car on the high­way is con­sid­ered “high-im­pact” is be­cause you are pe­nal­iz­ing pre­ven­tion as much as cau­sa­tion. Your sug­gested solu­tion of us­ing a sin­gle ac­tion to ac­ti­vate a self-driv­ing car for the whole high­way ride is clever, but has some prob­lems:

  • This greatly re­duces the gran­u­lar­ity of the penalty, mak­ing credit as­sign­ment more difficult.

  • This effec­tively uses the ini­tial-branch in­ac­tion baseline (branch­ing off when the self-driv­ing car is launched) in­stead of the step­wise in­ac­tion baseline, which means get­ting cling­i­ness is­sues back, in the sense of the agent be­ing pe­nal­ized for hu­man re­ac­tions to the self-driv­ing car.

  • You may not be able to pre­dict in ad­vance when the agent will en­counter situ­a­tions where the de­fault ac­tion is ir­re­versible or oth­er­wise un­de­sir­able.

  • In such situ­a­tions, the penalty will pro­duce bad in­cen­tives. Namely, the penalty for stay­ing on the road is pro­por­tionate to how bad a crash would be, so the trade­off with goal achieve­ment re­solves in an un­de­sir­able way. If we keep the re­ward for the car ar­riv­ing to its des­ti­na­tion con­stant, then as we in­crease the bad­ness of a crash (e.g. the num­ber of peo­ple on the side of the road who would be run over if the agent took a noop ac­tion), even­tu­ally the penalty wins in the trade­off with the re­ward, and the agent chooses the noop. I think it’s very im­por­tant to avoid this failure mode.

• Vika 23 Sep 2018 19:49 UTC

  6 points

  AF

  in reply to: TurnTrout's comment on: Towards a New Im­pact Measure

  Ac­tu­ally, I think it was in­cor­rect of me to frame this is­sue as a trade­off be­tween avoid­ing the sur­vival in­cen­tive and not crip­pling the agent’s ca­pa­bil­ity. What I was try­ing to point at is that the way you are coun­ter­act­ing the sur­vival in­cen­tive is by pe­nal­iz­ing the agent for in­creas­ing its power, and that in­terferes with the agent’s ca­pa­bil­ity. I think there may be other ways to coun­ter­act the sur­vival in­cen­tive with­out crip­pling the agent, and we should look for those first be­fore agree­ing to pay such a high price for in­ter­rupt­ibil­ity. I gen­er­ally be­lieve that ‘low im­pact’ is not the right thing to aim for, be­cause ul­ti­mately the goal of build­ing AGI is to have high im­pact—high benefi­cial im­pact. This is why I fo­cus on the op­por­tu­nity-cost-in­cur­ring as­pect of the prob­lem, i.e. avoid­ing side effects.

  Note that AUP could eas­ily be con­verted to a side-effects-only mea­sure by re­plac­ing the |differ­ence| with a max(0, differ­ence). Similarly, RR could be con­verted to a mea­sure that pe­nal­izes in­creases in power by do­ing the op­po­site (re­plac­ing max(0, differ­ence) with |differ­ence|). (I would ex­pect that var­i­ant of RR to coun­ter­act the sur­vival in­cen­tive, though I haven’t tested it yet.) Thus, it may not be nec­es­sary to re­solve the dis­agree­ment about whether it’s good to pe­nal­ize in­creases in power, since the same meth­ods can be adapted to both cases.

• Vika 20 Sep 2018 19:32 UTC

  3 points

  AF

  in reply to: TurnTrout's comment on: Towards a New Im­pact Measure

  If the agent isn’t over­com­ing ob­sta­cles, we can just in­crease N.

  Wouldn’t in­creas­ing N po­ten­tially in­crease the shut­down in­cen­tive, given the trade­off be­tween shut­down in­cen­tive and over­com­ing ob­sta­cles?

  I think elimi­nat­ing this sur­vival in­cen­tive is ex­tremely im­por­tant for this kind of agent, and ar­guably leads to be­hav­iors that are dras­ti­cally eas­ier to han­dle.

  I think we have a dis­agree­ment here about which desider­ata are more im­por­tant. Cur­rently I think it’s more im­por­tant for the im­pact mea­sure not to crip­ple the agent’s ca­pa­bil­ity, and the shut­down in­cen­tive might be eas­ier to coun­ter­act us­ing some more spe­cial­ized in­ter­rupt­ibil­ity tech­nique rather than an im­pact mea­sure. Not cer­tain about this though—I think we might need more ex­per­i­ments on more com­plex en­vi­ron­ments to get some idea of how bad this trade­off is in prac­tice.

  And why is this, given that the in­puts are his­to­ries? Why can’t we sim­ply mea­sure power?

  Your mea­sure­ment of “power” (I as­sume you mean Q_u?) needs to be grounded in the real world in some way. The ob­ser­va­tions will be raw pix­els or some­thing similar, while the util­ities and the en­vi­ron­ment model will be com­puted in terms of some sort of higher-level fea­tures or rep­re­sen­ta­tions. I would ex­pect the way these higher-level fea­tures are cho­sen or learned to af­fect the out­come of that com­pu­ta­tion.

  I dis­cussed in “Utility Selec­tion” and “AUP Un­bound” why I think this ac­tu­ally isn’t the case, sur­pris­ingly. What are your dis­agree­ments with my ar­gu­ments there?

  I found those sec­tions vague and un­clear (af­ter reread­ing a few times), and didn’t un­der­stand why you claim that a ran­dom set of util­ity func­tions would work. E.g. what do you mean by “long arms of op­por­tu­nity cost and in­stru­men­tal con­ver­gence”? What does the last para­graph of “AUP Un­bound” mean and how does it im­ply the claim?

  Oops, noted. I had a dis­tinct feel­ing of “if I’m go­ing to make claims this strong in a venue this crit­i­cal about a topic this im­por­tant, I bet­ter provide strong sup­port”.

  Pro­vid­ing strong sup­port is cer­tainly im­por­tant, but I think it’s more about clar­ity and pre­ci­sion than quan­tity. Bet­ter to give one clear sup­port­ing state­ment than many un­clear ones :).

• Vika 20 Sep 2018 16:26 UTC

  12 points

  AF

  on: Towards a New Im­pact Measure

  Great work! I like the ex­ten­sive set of desider­ata and test cases ad­dressed by this method.

  The biggest differ­ence from rel­a­tive reach­a­bil­ity, as I see it, is that you pe­nal­ize in­creas­ing the abil­ity to achieve goals, as well as de­creas­ing it. I’m not cur­rently sure whether this is a good idea: while it in­deed coun­ter­acts in­stru­men­tal in­cen­tives, it could also “crip­ple” the agent by in­cen­tiviz­ing it to set­tle for more sub­op­ti­mal solu­tions than nec­es­sary for safety.

  For ex­am­ple, the shut­down but­ton in the “sur­vival in­cen­tive” grid­world could be in­ter­preted as a su­per­vi­sor sig­nal (in which case the agent should not dis­able it) or as an ob­sta­cle in the en­vi­ron­ment (in which case the agent should dis­able it). Sim­ply pe­nal­iz­ing the agent for in­creas­ing its abil­ity to achieve goals leads to in­cor­rect be­hav­ior in the sec­ond case. To be­have cor­rectly in both cases, the agent needs more in­for­ma­tion about the source of the ob­sta­cle, which is not pro­vided in this grid­world (the Safe In­ter­rupt­ibil­ity grid­world has the same prob­lem).

  Another im­por­tant differ­ence is that you are us­ing a step­wise in­ac­tion baseline (branch­ing off at each time step rather than the ini­tial time step) and pre­dict­ing fu­ture effects us­ing an en­vi­ron­ment model. I think this is an im­prove­ment on the ini­tial-branch in­ac­tion baseline, which avoids cling­i­ness to­wards in­de­pen­dent hu­man ac­tions, but not to­wards hu­man re­ac­tions to the agent’s ac­tions. The en­vi­ron­ment model helps to avoid the is­sue with the step­wise in­ac­tion baseline failing to pe­nal­ize de­layed effects, though this will only pe­nal­ize de­layed effects if they are ac­cu­rately pre­dicted by the en­vi­ron­ment model (e.g. a de­layed effect that takes place be­yond the model’s plan­ning hori­zon will not be pe­nal­ized). I think the step­wise baseline + en­vi­ron­ment model could similarly be used in con­junc­tion with rel­a­tive reach­a­bil­ity.

  I agree with Char­lie that you are giv­ing out check­marks for the desider­ata a bit too eas­ily :). For ex­am­ple, I’m not con­vinced that your ap­proach is rep­re­sen­ta­tion-ag­nos­tic. It strongly de­pends on your choice of the set of util­ity func­tions and en­vi­ron­ment model, and those have to be ex­pressed in terms of the state of the world. (Note that the util­ity func­tions in your ex­am­ples, such as u_closet and u_left, are defined in terms of reach­ing a spe­cific state.) I don’t think your method can re­ally get away from mak­ing a choice of state rep­re­sen­ta­tion.

  Your ap­proach might have the same prob­lem as other value-ag­nos­tic ap­proaches (in­clud­ing rel­a­tive reach­a­bil­ity) with mostly pe­nal­iz­ing ir­rele­vant im­pacts. The AUP mea­sure seems likely to give most of its weight to util­ity func­tions that are ir­rele­vant to hu­mans, while the RR mea­sure could give most of its weight to pre­serv­ing reach­a­bil­ity of ir­rele­vant states. I don’t cur­rently know a way around this that’s not value-laden.

  Meta point: I think it would be valuable to have a more con­cise ver­sion of this post that in­tro­duces the key in­sight ear­lier on, since I found it a bit ver­bose and difficult to fol­low. The cur­rent writeup seems to be struc­tured ac­cord­ing to the or­der in which you gen­er­ated the ideas, rather than an or­der that would be more in­tu­itive to read­ers. FWIW, I had the same difficulty when writ­ing up the rel­a­tive reach­a­bil­ity pa­per, so I think it’s gen­er­ally challeng­ing to clearly pre­sent ideas about this prob­lem.