Vika

• Vika 18 Feb 2020 15:35 UTC

  LW: 4 AF: 2

  AF

  on: (In)ac­tion rollouts

  I don’t un­der­stand this pro­posal so far. I’m par­tic­u­larly con­fused by the last para­graph in the “to get away” sec­tion:

  • What does it mean in this con­text for A to im­ple­ment a policy? I thought A was build­ing a sub­agent and then fol­low­ing ${\pi }_0$ for­ever, thus not fol­low­ing ${\pi }_k^{\ast }$ for any $k$ at any point.

  • If A fol­lows ${\pi }_0$ for $\tau$ turns and then fol­lows ${\pi }_k^{\ast }$, how are $\tau$ and $k$ cho­sen?

  • It’s not clear to me that SA can act to en­sure the baseline value of $V_k^{\prime }$ for all val­ues of $k$ and $\tau$ un­less it does noth­ing.

  I think it might help to illus­trate this pro­posal in your origi­nal grid­world ex­am­ple to make it clearer what’s go­ing on. As far as I can tell so far, this does not ad­dress the is­sue I men­tioned ear­lier where if the sub­agent ac­tu­ally achieves any of the aux­iliary re­wards, sub­agent cre­ation will be pe­nal­ized.

• Vika 18 Feb 2020 12:24 UTC

  LW: 4 AF: 2

  AF

  in reply to: Stuart_Armstrong's comment on: Step­wise in­ac­tion and non-in­dex­i­cal im­pact measures

  I don’t think this re­quires iden­ti­fy­ing what a sub­agent is. You only need to be able to re­li­ably iden­tify the state be­fore the sub­agent is cre­ated (i.e. the start­ing state), but you don’t need to tell apart other states that are not the start­ing state.

  I agree that we need to com­pare to the penalty if the sub­agent is not cre­ated—I just wanted to show that sub­agent cre­ation does not avoid penalties. The penalty for sub­agent cre­ation will re­flect any im­pact the sub­agent ac­tu­ally causes in the en­vi­ron­ment (in the in­ac­tion rol­louts).

  As you men­tion in your other com­ment, cre­at­ing a sub­agent is effec­tively switch­ing from a step­wise in­ac­tion baseline to an in­ac­tion baseline for the rest of the epi­sode. This can be benefi­cial for the agent be­cause of the ‘wind­ing road’ prob­lem, where the step­wise baseline with in­ac­tion rol­louts can re­peat­edly pe­nal­ize ac­tions (e.g. turn­ing the wheel to stay on the road and avoid crash­ing) that are not pe­nal­ized by the in­ac­tion baseline. This is a gen­eral is­sue with in­ac­tion rol­louts that needs to be fixed.

• Vika 17 Feb 2020 22:11 UTC

  LW: 6 AF: 3

  AF

  on: Step­wise in­ac­tion and non-in­dex­i­cal im­pact measures

  I think this prob­lem is about cap­tur­ing de­layed effects of the agent’s ac­tions. The way the step­wise baseline is sup­posed to pe­nal­ize de­layed effects is us­ing in­ac­tion rol­louts, which com­pare the effects of the agent ac­tion + k noops and the effects of k+1 noops (for all pos­i­tive in­te­gers k). I don’t think it’s use­ful to con­sider this is­sue for a step­wise baseline with no rol­louts, since that baseline fails to pe­nal­ize any de­layed effects (no sub­agents needed).

  Here, the in­ac­tion rol­louts don’t cap­ture the effects of the sub­agent, and thus the agent doesn’t re­ceive a penalty for cre­at­ing the sub­agent. There are sev­eral fac­tors that con­tribute to this:

  1. The aux­iliary re­ward R only re­wards the agent it­self press­ing the red but­ton, not the red but­ton be­ing pressed in gen­eral, so it would not be trig­gered by the sub­agent press­ing it.

  2. The sub­agent never goes to the red but­ton—it only goes to the blue but­ton.

  The agent would be pe­nal­ized for cre­at­ing the sub­agent if both 1 and 2 were false. If 1 was false, the aux­iliary re­ward would be given when­ever the red but­ton was pressed. If 2 was also false, the in­ac­tion rol­lout af­ter sub­agent cre­ation would in­clude a state where the sub­agent presses the red but­ton, and so the agent would be pe­nal­ized.

  In the gen­eral case, the agent can only get away with cre­at­ing sub­agents that don’t achieve any aux­iliary re­wards, which I think limits the mag­ni­tude of this prob­lem. The de­signer can choose aux­iliary re­wards (that are not achieved by the agent do­ing noth­ing) such that a sub­agent will achieve at least one of them. For ex­am­ple, if there is an aux­iliary re­ward for reach­ing any state ex­cept the state be­fore sub­agent cre­ation, the no-sub­agent in­ac­tion rol­lout will con­sist of this state, and the sub­agent in­ac­tion rol­lout will con­tain a differ­ent state at least once, so sub­agent cre­ation will be pe­nal­ized.

• Vika 13 Feb 2020 15:10 UTC

  LW: 6 AF: 3

  AF

  on: Im­pact mea­sures and subagents

  Thanks Stu­art for your thought-pro­vok­ing post! I think your point about the effects of the baseline choice on the sub­agent prob­lem is very in­ter­est­ing, and it would be helpful to sep­a­rate it more clearly from the effects of the de­vi­a­tion mea­sure (which are cur­rently a bit con­flated in the table). I ex­pect that AU with the in­ac­tion baseline would also avoid this is­sue, similarly to RR with an in­ac­tion baseline. I sus­pect that the twenty billion ques­tions mea­sure with the step­wise baseline would have the sub­agent is­sue too.

  I’m won­der­ing whether this is­sue is en­tirely caused by the step­wise baseline (which is in­dexed on the agent, as you point out), or whether the op­tion­al­ity-based de­vi­a­tion mea­sures (RR and AU) con­tribute to it as well. So far I’m adding this to my men­tal list of is­sues with the step­wise baseline (along with the “car on a wind­ing road” sce­nario) that need to be fixed.

  What links here?

  • Step­wise in­ac­tion and non-in­dex­i­cal im­pact measures by Stuart_Armstrong (17 Feb 2020 10:32 UTC; 12 points)

• Vika 20 Dec 2019 16:22 UTC

  LW: 31 AF: 12

  AF

  on: Speci­fi­ca­tion gam­ing ex­am­ples in AI

  I’ve been pleas­antly sur­prised by how much this re­source has caught on in terms of peo­ple us­ing it and refer­ring to it (definitely more than I ex­pected when I made it). There were 30 ex­am­ples on the list when was posted in April 2018, and 20 new ex­am­ples have been con­tributed through the form since then. I think the list has sev­eral prop­er­ties that con­tributed to wide adop­tion: it’s fun, stan­dard­ized, up-to-date, com­pre­hen­sive, and col­lab­o­ra­tive.

  Some of the ap­peal is that it’s fun to read about AI cheat­ing at tasks in un­ex­pected ways (I’ve seen a lot of peo­ple post on Twit­ter about their fa­vorite ex­am­ples from the list). The stan­dard­ized spread­sheet for­mat seems eas­ier to re­fer to as well. I think the crowd­sourc­ing as­pect is also helpful—this helps keep it cur­rent and com­pre­hen­sive, and peo­ple can feel some own­er­ship of the list since can per­son­ally con­tribute to it. My over­all take­away from this is that safety out­reach tools are more likely to be im­pact­ful if they are fun and easy for peo­ple to en­gage with.

  This list had a sur­pris­ing amount of im­pact rel­a­tive to how lit­tle work it took me to put it to­gether and main­tain it. The hard work of find­ing and sum­ma­riz­ing the ex­am­ples was done by the peo­ple putting to­gether the lists that the mas­ter list draws on (Gw­ern, Lehman, Ols­son, Ir­pan, and oth­ers), as well as the peo­ple who sub­mit ex­am­ples through the form. What I do is put them to­gether in a com­mon for­mat and clar­ify and/​or shorten some of the sum­maries. I also cu­rate the ex­am­ples to de­ter­mine whether they fit the defi­ni­tion of speci­fi­ca­tion gam­ing (as op­posed to sim­ply a sur­pris­ing be­hav­ior or solu­tion). Over­all, I’ve prob­a­bly spent around 10 hours so far on cre­at­ing and main­tain­ing the list, which is not very much. This makes me won­der if there is other low hang­ing fruit in the safety re­sources space that we haven’t picked yet.

  I have been us­ing it both as an out­reach and re­search tool. On the out­reach side, the re­source has been helpful for mak­ing the ar­gu­ment that safety prob­lems are hard and need gen­eral solu­tions, by mak­ing it salient just in how many ways things could go wrong. When pre­sented with an in­di­vi­d­ual ex­am­ple of speci­fi­ca­tion gam­ing, peo­ple of­ten have a de­fault re­ac­tion of “well, you can just close the loop­hole like this”. It’s eas­ier to see that this ap­proach does not scale when pre­sented with 50 ex­am­ples of gam­ing be­hav­iors. Any given loop­hole can seem ob­vi­ous in hind­sight, but 50 loop­holes are much less so. I’ve found this use­ful for com­mu­ni­cat­ing a sense of the difficulty and im­por­tance of Good­hart’s Law.

  On the re­search side, the ex­am­ples have been helpful for try­ing to clar­ify the dis­tinc­tion be­tween re­ward gam­ing and tam­per­ing prob­lems. Re­ward gam­ing hap­pens when the re­ward func­tion is de­signed in­cor­rectly (so the agent is gam­ing the de­sign speci­fi­ca­tion), while re­ward tam­per­ing hap­pens when the re­ward func­tion is im­ple­mented in­cor­rectly or em­bed­ded in the en­vi­ron­ment (and so can be thought of as gam­ing the im­ple­men­ta­tion speci­fi­ca­tion). The boat race ex­am­ple is re­ward gam­ing, since the score func­tion was defined in­cor­rectly, while the Qbert agent find­ing a bug that makes the plat­forms blink and gives the agent mil­lions of points is re­ward tam­per­ing. We don’t cur­rently have any real ex­am­ples of the agent gain­ing con­trol of the re­ward chan­nel (prob­a­bly be­cause the ac­tion spaces of pre­sent-day agents are too limited), which seems qual­i­ta­tively differ­ent from the nu­mer­ous ex­am­ples of agents ex­ploit­ing im­ple­men­ta­tion bugs.

  I’m cu­ri­ous what peo­ple find the list use­ful for—as a safety out­reach tool, a re­search tool or in­tu­ition pump, or some­thing else? I’d also be in­ter­ested in sug­ges­tions for im­prov­ing the list (for­mat­ting, cat­e­go­riz­ing, etc). Thanks ev­ery­one who has con­tributed to the re­source so far!

• Vika 17 Dec 2019 13:53 UTC

  LW: 3 AF: 2

  AF

  in reply to: Ben Pace's comment on: Speci­fi­ca­tion gam­ing ex­am­ples in AI

  Thanks Ben! I’m happy that the list has been a use­ful re­source. A lot of credit goes to Gw­ern, who col­lected many ex­am­ples that went into the speci­fi­ca­tion gam­ing list: https://​​www.gw­ern.net/​​Tanks#al­ter­na­tive-ex­am­ples.

• Vika 21 Oct 2019 14:55 UTC

  LW: 5 AF: 2

  AF

  in reply to: cousin_it's comment on: Thoughts on “Hu­man-Com­pat­i­ble”

  Yes, de­cou­pling seems to ad­dress a broad class of in­cen­tive prob­lems in safety, which in­cludes the shut­down prob­lem and var­i­ous forms of tam­per­ing /​ wire­head­ing. Other ex­am­ples of de­cou­pling in­clude causal coun­ter­fac­tual agents and coun­ter­fac­tual re­ward mod­el­ing.

• Vika 29 Aug 2019 11:03 UTC

  LW: 9 AF: 5

  AF

  in reply to: evhub's comment on: Clas­sify­ing speci­fi­ca­tion prob­lems as var­i­ants of Good­hart’s Law

  Thanks Evan, glad you found this use­ful! The con­nec­tion with the in­ner/​outer al­ign­ment dis­tinc­tion seems in­ter­est­ing. I agree that the in­ner al­ign­ment prob­lem falls in the de­sign-emer­gent gap. Not sure about the outer al­ign­ment prob­lem match­ing the ideal-de­sign gap though, since I would clas­sify tam­per­ing prob­lems as outer al­ign­ment prob­lems, caused by flaws in the im­ple­men­ta­tion of the base ob­jec­tive.

• Vika 29 Aug 2019 10:50 UTC

  LW: 24 AF: 9

  AF

  in reply to: TurnTrout's comment on: Rev­ersible changes: con­sider a bucket of water

  I think the dis­cus­sion of re­versibil­ity and molecules is a dis­trac­tion from the core of Stu­art’s ob­jec­tion. I think he is say­ing that a value-ag­nos­tic im­pact mea­sure can­not dis­t­in­guish be­tween the cases where the wa­ter in the bucket is or isn’t valuable (e.g. whether it has sen­ti­men­tal value to some­one).

  If AUP is not value-ag­nos­tic, it is us­ing hu­man prefer­ence in­for­ma­tion to fill in the “what we want” part of your defi­ni­tion of im­pact, i.e. define the aux­iliary util­ity func­tions. In this case I would ex­pect you and Stu­art to be in agree­ment.

  If AUP is value-ag­nos­tic, it is not us­ing hu­man prefer­ence in­for­ma­tion. Then I don’t see how the state rep­re­sen­ta­tion/​on­tol­ogy in­var­i­ance prop­erty helps to dis­t­in­guish be­tween the two cases. As dis­cussed in this com­ment, state rep­re­sen­ta­tion in­var­i­ance holds over all rep­re­sen­ta­tions that are con­sis­tent with the true hu­man re­ward func­tion. Thus, you can dis­t­in­guish the two cases as long as you are us­ing one of these re­ward-con­sis­tent rep­re­sen­ta­tions. How­ever, since a value-ag­nos­tic im­pact mea­sure does not have ac­cess to the true re­ward func­tion, you can­not guaran­tee that the state rep­re­sen­ta­tion you are us­ing to com­pute AUP is in the re­ward-con­sis­tent set. Then, you could fail to dis­t­in­guish be­tween the two cases, giv­ing the same penalty for kick­ing a more or less valuable bucket.

• Vika 28 Aug 2019 11:40 UTC

  LW: 6 AF: 3

  AF

  on: Rev­ersible changes: con­sider a bucket of water

  Thanks Stu­art for the ex­am­ple. There are two ways to dis­t­in­guish the cases where the agent should and shouldn’t kick the bucket:

  • Rel­a­tive value of the bucket con­tents com­pared to the goal is rep­re­sented by the weight on the im­pact penalty rel­a­tive to the re­ward. For ex­am­ple, if the agent’s goal is to put out a fire on the other end of the pool, you would set a low weight on the im­pact penalty, which en­ables the agent to take ir­re­versible ac­tions in or­der to achieve the goal. This is why im­pact mea­sures use a re­ward-penalty trade­off rather than a con­straint on ir­re­versible ac­tions.

  • Ab­solute value of the bucket con­tents can be rep­re­sented by adding weights on the reach­able states or at­tain­able util­ity func­tions. This doesn’t nec­es­sar­ily re­quire defin­ing hu­man prefer­ences or pro­vid­ing hu­man in­put, since hu­man prefer­ences can be in­ferred from the start­ing state. I gen­er­ally think that im­pact mea­sures don’t have to be value-ag­nos­tic, as long as they re­quire less in­put about hu­man prefer­ences than the gen­eral value learn­ing prob­lem.

Clas­sify­ing speci­fi­ca­tion prob­lems as var­i­ants of Good­hart’s Law

• Vika 19 Aug 2019 14:23 UTC

  LW: 6 AF: 3

  AF

  on: Stable Poin­t­ers to Value: An Agent Embed­ded in Its Own Utility Function

  Thanks Abram for this se­quence—for some rea­son I wasn’t aware of it un­til some­one linked to it re­cently.

  Would you con­sider the ob­ser­va­tion tam­per­ing (delu­sion box) prob­lem as part of the easy prob­lem, the hard prob­lem, or a differ­ent prob­lem al­to­gether? I think it must be a differ­ent prob­lem, since it is not ad­dressed by ob­ser­va­tion-util­ity or ap­proval-di­rec­tion.

• Vika 22 Jul 2019 19:46 UTC

  50 points

  in reply to: paulfchristiano's comment on: The AI Timelines Scam

  Definitely agree that the AI com­mu­nity is not bi­ased to­wards short timelines. Long timelines are the dom­i­nant view, while the short timelines view is as­so­ci­ated with hype. Many re­searchers are con­cerned about the field los­ing cred­i­bil­ity (and fund­ing) if the hype bub­ble bursts, and this is es­pe­cially true for those who ex­pe­rienced the AI win­ters. They see the long timelines view as ap­pro­pri­ately skep­ti­cal and more sci­en­tifi­cally re­spectable.

  Some ex­am­ples of state­ments that AGI is far away from high-pro­file AI re­searchers:

  Ge­offrey Hin­ton: https://​​ven­ture­beat.com/​​2018/​​12/​​17/​​ge­offrey-hin­ton-and-demis-has­s­abis-agi-is-nowhere-close-to-be­ing-a-re­al­ity/​​

  Yann LeCun: https://​​www.face­book.com/​​yann.le­cun/​​posts/​​10153426023477143 https://​​fu­tur­ism.com/​​con­scious-ai-decades-away https://​​www.face­book.com/​​yann.le­cun/​​posts/​​10153368458167143

  Yoshua Ben­gio: https://​​www.less­wrong.com/​​posts/​​4qPy8jwRxLg9qWLiG/​​yoshua-ben­gio-on-ai-progress-hype-and-risks

  Rod­ney Brooks: https://​​rod­ney­brooks.com/​​the-seven-deadly-sins-of-pre­dict­ing-the-fu­ture-of-ai/​​ https://​​rod­ney­brooks.com/​​agi-has-been-de­layed/​​

• Vika 6 Jul 2019 10:31 UTC

  LW: 7 AF: 4

  AF

  in reply to: RyanCarey's comment on: TAISU—Tech­ni­cal AI Safety Unconference

  Janos and I are com­ing for the week­end part of the unconference

• Vika 3 Jul 2019 13:55 UTC

  LW: 10 AF: 5

  AF

  on: Risks from Learned Op­ti­miza­tion: Introduction

  I’m con­fused about the differ­ence be­tween a mesa-op­ti­mizer and an emer­gent sub­agent. A “par­tic­u­lar type of al­gorithm that the base op­ti­mizer might find to solve its task” or a “neu­ral net­work that is im­ple­ment­ing some op­ti­miza­tion pro­cess” in­side the base op­ti­mizer seem like emer­gent sub­agents to me. What is your defi­ni­tion of an emer­gent sub­agent?

• Vika 11 May 2019 3:50 UTC

  LW: 6 AF: 3

  AF

  in reply to: rohinmshah's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  Thanks Ro­hin! Your ex­pla­na­tions (both in the com­ments and offline) were very helpful and clar­ified a lot of things for me. My cur­rent un­der­stand­ing as a re­sult of our dis­cus­sion is as fol­lows.

  AU is a func­tion of the world state, but in­tends to cap­ture some gen­eral mea­sure of the agent’s in­fluence over the en­vi­ron­ment that does not de­pend on the state rep­re­sen­ta­tion.

  Here is a hi­er­ar­chy of ob­jects, where each ob­ject is a func­tion of the pre­vi­ous one: world states /​ microstates (e.g. quark con­figu­ra­tion) → ob­ser­va­tions (e.g. pix­els) → state rep­re­sen­ta­tion /​ coarse-grain­ing (which defines macrostates as equiv­alence classes over ob­ser­va­tions) → fea­tur­iza­tion (a coarse-grain­ing that fac­tor­izes into fea­tures). The im­pact mea­sure is defined over the macrostates.

  Con­sider the set of all state rep­re­sen­ta­tions that are con­sis­tent with the true re­ward func­tion (i.e. if two microstates have differ­ent true re­wards, then their state rep­re­sen­ta­tion is differ­ent). The im­pact mea­sure is rep­re­sen­ta­tion-in­var­i­ant if it has the same val­ues for any state rep­re­sen­ta­tion in this re­ward-com­pat­i­ble set. (Note that if rep­re­sen­ta­tion in­var­i­ance was defined over the set of all pos­si­ble state rep­re­sen­ta­tions, this set would in­clude the most coarse-grained rep­re­sen­ta­tion with all ob­ser­va­tions in one macrostate, which would im­ply that the im­pact mea­sure is always 0.) Now con­sider the most coarse-grained rep­re­sen­ta­tion R that is con­sis­tent with the true re­ward func­tion.

  An AU mea­sure defined over R would re­main the same for a finer-grained rep­re­sen­ta­tion. For ex­am­ple, if the at­tain­able set con­tains a re­ward func­tion that re­wards hav­ing a vase in the room, and the rep­re­sen­ta­tion is re­fined to dis­t­in­guish green and blue vases, then macrostates with differ­ent-col­ored vases would re­ceive the same re­ward. Thus, this mea­sure would be rep­re­sen­ta­tion-in­var­i­ant. How­ever, for an AU mea­sure defined over a finer-grained rep­re­sen­ta­tion (e.g. dis­t­in­guish­ing blue and green vases), a ran­dom re­ward func­tion in the at­tain­able set could as­sign a differ­ent re­ward to macrostates with blue and green vases, and the re­sult­ing mea­sure would be differ­ent from the mea­sure defined over R.

  An RR mea­sure that only uses reach­a­bil­ity func­tions of sin­gle macrostates is not rep­re­sen­ta­tion-in­var­i­ant, be­cause the ob­ser­va­tions in­cluded in each macrostate de­pend on the coarse-grain­ing. How­ever, if we al­low the RR mea­sure to use reach­a­bil­ity func­tions of sets of macrostates, then it would be rep­re­sen­ta­tion-in­var­i­ant if it is defined over R. Then a func­tion that re­wards reach­ing a macrostate with a vase can be defined in a finer-grained rep­re­sen­ta­tion by re­ward­ing macrostates with green or blue vases. Thus, both AU and this ver­sion of RR are rep­re­sen­ta­tion-in­var­i­ant iff they are defined over the most coarse-grained rep­re­sen­ta­tion con­sis­tent with the true re­ward.

  What links here?

  • Vika's comment on Rev­ersible changes: con­sider a bucket of water by Stuart_Armstrong (29 Aug 2019 10:50 UTC; 24 points)

• Vika 3 May 2019 13:44 UTC

  LW: 6 AF: 2

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  There are var­i­ous parts of your ex­pla­na­tion that I find vague and could use a clar­ifi­ca­tion on:

  • “AUP is not about state”—what does it mean for a method to be “about state”? Same goes for “the di­rect fo­cus should not be on the state”—what does “di­rect fo­cus” mean here?

  • “Overfit­ting the en­vi­ron­ment”—I know what it means to overfit a train­ing set, but I don’t know what it means to overfit an en­vi­ron­ment.

  • “The long arms of op­por­tu­nity cost and in­stru­men­tal con­ver­gence”—what do “long arms” mean?

  • “Wire­head a util­ity func­tion”—is this the same as op­ti­miz­ing a util­ity func­tion?

  • “Cut out the mid­dle­man”—what are you refer­ring to here?

  I think these in­tu­itive phrases may be a use­ful short­hand for some­one who already un­der­stands what you are talk­ing about, but since I do not un­der­stand, I have not found them illu­mi­nat­ing.

  I sym­pa­thize with your frus­tra­tion about the difficulty of com­mu­ni­cat­ing these com­plex ideas clearly. I think the difficulty is caused by the vague lan­guage rather than miss­ing key ideas, and mak­ing the lan­guage more pre­cise would go a long way.

• Vika 2 May 2019 17:01 UTC

  LW: 6 AF: 3

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  Thanks for the de­tailed ex­pla­na­tion—I feel a bit less con­fused now. I was not in­tend­ing to ex­press con­fi­dence about my pre­dic­tion of what AU does. I was aware that I didn’t un­der­stand the state rep­re­sen­ta­tion in­var­i­ance claim in the AUP pro­posal, though I didn’t re­al­ize that it is as cen­tral to the pro­posal as you de­scribe here.

  I am still con­fused about what you means by pe­nal­iz­ing ‘power’ and what ex­actly it is a func­tion of. The way you de­scribe it here sounds like it’s a mea­sure of the agent’s op­ti­miza­tion abil­ity that does not de­pend on the state at all. Did you mean that in the real world the agent always re­ceives the same AUP penalty no mat­ter which state it is in? If that is what you meant, then I’m not sure how to rec­on­cile your de­scrip­tion of AUP in the real world (where the penalty is not a func­tion of the state) and AUP in an MDP (where it is a func­tion of the state). I would find it helpful to see a defi­ni­tion of AUP in a POMDP as an in­ter­me­di­ate case.

  I agree with Daniel’s com­ment that if AUP is not pe­nal­iz­ing effects on the world, then it is con­fus­ing to call it an ‘im­pact mea­sure’, and some­thing like ‘op­ti­miza­tion reg­u­lariza­tion’ would be bet­ter.

  Since I still have lin­ger­ing con­fu­sions af­ter your lat­est ex­pla­na­tion, I would re­ally ap­pre­ci­ate if some­one else who un­der­stands this could ex­plain it to me.

• Vika 22 Apr 2019 17:36 UTC

  LW: 2 AF: 1

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  Are you think­ing of an ac­tion ob­ser­va­tion for­mal­ism, or some kind of re­ward func­tion over in­ferred state?

  I don’t quite un­der­stand what you’re ask­ing here, could you clar­ify?

  If you had to pose the prob­lem of im­pact mea­sure­ment as a ques­tion, what would it be?

  Some­thing along the lines of: “How can we mea­sure to what ex­tent the agent is chang­ing the world in ways that we care about?”. Why?

• Vika 20 Apr 2019 13:23 UTC

  LW: 2 AF: 1

  AF

  in reply to: TurnTrout's comment on: Best rea­sons for pes­simism about im­pact of im­pact mea­sures?

  What does this mean, con­cretely? And what hap­pens with the sur­vival util­ity func­tion be­ing the sole mem­ber of the at­tain­able set? Does this run into that prob­lem, in your model?

  I meant that for at­tain­able set con­sist­ing of ran­dom util­ity func­tions, I would ex­pect most of the vari­a­tion in util­ity to be based on ir­rele­vant fac­tors like the po­si­tions of air molecules. This does not ap­ply to the at­tain­able set con­sist­ing of the sur­vival util­ity func­tion, since that is not a ran­dom util­ity func­tion.

  What makes you think that?

  This is an in­tu­itive claim based on a gen­eral ob­ser­va­tion of how peo­ple at­tribute re­spon­si­bil­ity. For ex­am­ple, if I walk into a busy street and get hit by a car, I will be con­sid­ered re­spon­si­ble for this be­cause it’s easy to pre­dict. On the other hand, if I am walk­ing down the street and a brick falls on my head from the nearby build­ing, then I will not be con­sid­ered re­spon­si­ble, be­cause this event would be hard to pre­dict. There are prob­a­bly other rea­sons that hu­mans don’t con­sider them­selves re­spon­si­ble for but­terfly effects.