TurnTrout

• TurnTrout 18 Feb 2020 18:34 UTC

  4 points

  in reply to: Pattern's comment on: At­tain­able Utility Preser­va­tion: Concepts

  Thanks for do­ing this. I was origi­nally go­ing to keep a text ver­sion of the whole se­quence, but I ended up mak­ing lots of fi­nal ed­its in the images, and this se­quence has already taken an in­cred­ible amount of time on my part.

• TurnTrout 18 Feb 2020 18:32 UTC

  LW: 2 AF: 1

  AF

  in reply to: Pattern's comment on: At­tain­able Utility Preser­va­tion: Concepts

  if we make sure that power is low enough we can turn it off, if the agent will ac­quire power if that’s the only way to achieve its goal rather than stop­ping at/​be­fore some limit then it might still ac­quire power and be catas­trophic*, etc.

  Yeah. I have the math for this kind of trade­off worked out—stay tuned!

  Though fur­ther up this com­ment I brought up the pos­si­bil­ity that “power seek­ing be­hav­ior is the cause of catas­tro­phe, rather than hav­ing power.”

  I think this is true, ac­tu­ally; if an­other agent already has a lot of power and it isn’t already catas­trophic for us, their con­tinued ex­is­tence isn’t that big of a deal wrt the sta­tus quo. The bad stuff comes with the change in who has power.

  The act of tak­ing away our power is gen­er­ally only in­cen­tivized so the agent can be­come bet­ter able to achieve its own goal. The ques­tion is, why is the agent try­ing to con­vince us of some­thing /​ get some­one else to do some­thing catas­trophic, if the agent isn’t try­ing to in­crease its own AU?

• TurnTrout 18 Feb 2020 15:15 UTC

  2 points

  in reply to: whpearson's comment on: At­tain­able Utility Preser­va­tion: Concepts

  The power limi­ta­tion isn’t a hard cap, it’s a trade­off. AUP agents do not have to half-ass any­thing. As I wrote in an­other com­ment,

  It prefers plans that don’t gain un­nec­es­sary power.

  If “un­nec­es­sary” is too squishy of a word for your tastes, I’m go­ing get quite spe­cific in the next few posts.

• TurnTrout 18 Feb 2020 3:14 UTC

  LW: 4 AF: 2

  AF

  in reply to: Pattern's comment on: At­tain­able Utility Preser­va­tion: Concepts

  The con­clu­sion doesn’t fol­low from the premise.

  CCC says (for non-evil goals) “if the op­ti­mal policy is catas­trophic, then it’s be­cause of power-seek­ing”. So its con­tra­pos­i­tive is in­deed as stated.

  Note that pre­serv­ing our at­tain­able util­ities isn’t a good thing, it’s just not a bad thing.

  I meant “pre­serv­ing” as in “not in­cen­tivized to take away power from us”, not “keeps us from benefit­ting from any­thing”, but you’re right about the im­pli­ca­tion as stated. Sorry for the am­bi­guity.

  Is this a metaphor for mak­ing an ‘agent’ with that goal, or ac­tu­ally cre­at­ing an agent that we can give differ­ent com­mands to and switch out/​mod­ify/​add to its goals?

  Me­taphor.

  “AUP_con­cep­tual solves this “lo­cal­ity” prob­lem by reg­u­lariz­ing the agent’s im­pact on the nearby AU land­scape.”

  Nearby from its per­spec­tive? (From a prac­ti­cal stand­point, if you’re close to an air­port you’re close to a lot of places on earth, that you aren’t from a ‘space’ per­spec­tive.)

  Nearby wrt this kind of “AU dis­tance/​prac­ti­cal per­spec­tive”, yes. Great catch.

  Also the agent might be con­cerned with flows rather than ac­tions.* We have an in­tu­itive no­tion that ‘build­ing fac­to­ries in­creases power’, but what about redi­rect­ing a river/​stream/​etc. with dams or dig­ging new paths for wa­ter to flow? What does the agent do if it un­ex­pect­edly gains power by some means, or re­al­izes its pa­per­clip ma­chines can be used to move straw­ber­ries/​make a copy it­self which is weaker but less con­strained? Can the agent make a ma­chine that makes pa­per­clips/​make mak­ing pa­per­clips eas­ier?

  As a con­se­quence of this be­ing a more effec­tive ap­proach—it makes cer­tain im­prove­ments ob­vi­ous. If you have a re­ally long com­mute to work, you might wish you lived closer to your work. (You might also be aware that houses closer to your work are more ex­pen­sive, but hu­mans are good at pick­ing up on this kind of low hang­ing fruit. A ca­pa­ble agent that thinks about pro­cess see­ing ‘op­por­tu­ni­ties to gain power’ is of some gen­eral con­cern. In this case be­cause an agent that tries to min­i­mize re­duc­ing/​af­fect­ing* other agents at­tain­able util­ity, with­out know­ing/​need­ing to know about other agents is some­what coun­ter­in­tu­itive.

  **It’s not clear if in­creas­ing shows up on the AUP map, or how that’s han­dled.

  Great thoughts. I think some of this will be an­swered in a few posts by the spe­cific im­ple­men­ta­tion de­tails. What do you mean by “AUP map”? The AU land­scape?

  What does the agent do if it un­ex­pect­edly gains power by some means,

  The idea is it only pe­nal­izes ex­pected power gain.

• TurnTrout 17 Feb 2020 17:25 UTC

  LW: 4 AF: 2

  AF

  in reply to: Kaj_Sotala's comment on: The Catas­trophic Con­ver­gence Conjecture

  In­trigu­ing. I don’t know whether that sug­gests our val­ues aren’t as com­pli­cated as we thought, or whether the pres­sures which se­lected them are not com­pli­cated.

  While I’m not an ex­pert on the biolog­i­cal in­trin­sic mo­ti­va­tion liter­a­ture, I think it’s at least true that some parts of our val­ues were se­lected for be­cause they’re good heuris­tics for main­tain­ing AU. This is the thing that MCE was try­ing to ex­plain:

  The pa­per’s cen­tral no­tion be­gins with the claim is that there is a phys­i­cal prin­ci­ple, called “causal en­tropic forces,” that drives a phys­i­cal sys­tem to­ward a state that max­i­mizes its op­tions for fu­ture change. For ex­am­ple, a par­ti­cle in­side a rec­t­an­gu­lar box will move to the cen­ter rather than to the side, be­cause once it is at the cen­ter it has the op­tion of mov­ing in any di­rec­tion. More­over, ar­gues the pa­per, phys­i­cal sys­tems gov­erned by causal en­tropic forces ex­hibit in­tel­li­gent be­hav­ior.

  I think they have this back­wards: in­tel­li­gent be­hav­ior of­ten re­sults in in­stru­men­tally con­ver­gent be­hav­ior (and not nec­es­sar­ily the other way around). Similarly, Salge et al. overview the be­hav­ioral em­pow­er­ment hy­poth­e­sis:

  The adap­ta­tion brought about by nat­u­ral evolu­tion re­duce or­ganisms that in ab­sence of spe­cific goals be­have as if they were max­i­miz­ing [mu­tual in­for­ma­tion be­tween their ac­tions and fu­ture ob­ser­va­tions].

  As I dis­cuss in sec­tion 6.1 of Op­ti­mal Far­sighted Agents Tend to Seek Power, I think that “abil­ity to achieve goals in gen­eral” (power) is a bet­ter in­tu­itive and tech­ni­cal no­tion than in­for­ma­tion-the­o­retic em­pow­er­ment. I think it’s pretty plau­si­ble that we have heuris­tics which, all else equal, push us to main­tain or in­crease our power.

• TurnTrout 17 Feb 2020 16:42 UTC

  2 points

  in reply to: Gurkenglas's comment on: At­tain­able Utility Preser­va­tion: Concepts

  This post is about AUP-the-con­cept, not about spe­cific im­ple­men­ta­tions. That plan in­creases its abil­ity to have pa­per­clips max­i­mized and so is pe­nal­ized by AUP${}_{\text{conceptual}}$. We’ll talk speci­fics later.

  ETA: As a more gen­eral note, this post should definitely have an “aha!” as­so­ci­ated with it, so if it doesn’t, I en­courage peo­ple to ask ques­tions.

• TurnTrout 17 Feb 2020 15:41 UTC

  LW: 2 AF: 1

  AF

  on: Subagents and im­pact mea­sures: sum­mary tables

  RR at­tempted to con­trol the side-effects of an agent by en­sur­ing it had enough power to reach a lot of states; this effect is not neu­tral­ised by a sub­agent.

  Things might get com­pli­cated by par­tial ob­serv­abil­ity; in the real world, the agent is min­i­miz­ing change in its be­liefs about what it can reach. Other­wise, you could just get around the SA prob­lem for AUP as well by sub­sti­tut­ing the re­ward func­tions for state in­di­ca­tor re­ward func­tions.

• TurnTrout 17 Feb 2020 15:32 UTC

  LW: 2 AF: 1

  AF

  on: Step­wise in­ac­tion and non-in­dex­i­cal im­pact measures

  I’ll es­tab­lish two facts: that un­der the step­wise in­ac­tion baseline, a sub­agent com­pletely un­der­mines all im­pact mea­sures (in­clud­ing twenty billion ques­tions).

  Note this im­plic­itly as­sumes an agent benefits by build­ing the sub­agent. The spe­cific coun­terex­am­ple I have in mind will be a few posts later in my se­quence.

• TurnTrout 17 Feb 2020 15:21 UTC

  LW: 5 AF: 3

  AF

  in reply to: steve2152's comment on: At­tain­able Utility Preser­va­tion: Concepts

  Depends how much power that gains com­pared to other plans. It prefers plans that don’t gain un­nec­es­sary power.

  In fact, the “en­couraged policy” in the post has the agent read­ing a Paper­clips for Dum­mies book and mak­ing a few ex­tra pa­per­clips.

At­tain­able Utility Preser­va­tion: Concepts

• TurnTrout 17 Feb 2020 5:20 UTC

  LW: 2 AF: 1

  AF

  in reply to: Charlie Steiner's comment on: The Catas­trophic Con­ver­gence Conjecture

  I’m not think­ing of op­ti­miz­ing for “not an ob­jec­tive catas­tro­phe” di­rectly—it’s just a use­ful con­cept. The next post cov­ers this.

• TurnTrout 15 Feb 2020 23:52 UTC

  LW: 3 AF: 2

  AF

  in reply to: steve2152's comment on: The Catas­trophic Con­ver­gence Conjecture

  Note that when I said

  (we don’t need any fancy en­g­ineer­ing or ar­bi­trary choices to figure out AUs/​op­ti­mal value from the agent’s per­spec­tive).

  I meant we could just con­sider how the agent’s AUs are chang­ing with­out lo­cat­ing a hu­man in the en­vi­ron­ment.

• TurnTrout 15 Feb 2020 14:17 UTC

  LW: 2 AF: 1

  AF

  in reply to: steve2152's comment on: The Catas­trophic Con­ver­gence Conjecture

  Wouldn’t you need to lo­cate the ab­stract con­cept of AU within the AI’s on­tol­ogy? Is that eas­ier? Or sorry if I’m mi­s­un­der­stand­ing.

  To the con­trary, an AU is nat­u­rally calcu­lated from re­ward, one of the few things that is on­tolog­i­cally fun­da­men­tal in the paradigm of RL. As men­tioned in the last post, the AU of re­ward func­tion $R$ is $V_R^{\ast }$ - which calcu­lates the max­i­mum pos­si­ble $R$-re­turn from a given state.

  This will be­come much more ob­vi­ous in the AUP em­piri­cal post.

• TurnTrout 15 Feb 2020 5:19 UTC

  3 points

  in reply to: Andrew Jacob Sauer's comment on: The Catas­trophic Con­ver­gence Conjecture

  Cal­ling both of these things a “catas­tro­phe” seems to sweep that differ­ence un­der the rug.

  Sure, but just like it makes sense to be able to say that a class of out­comes is “good” with­out ev­ery sin­gle such out­come be­ing max­i­mally good, it makes sense to have a con­cept for catas­tro­phes, even if they’re not liter­ally the worst things pos­si­ble.

  Which seems bound to hap­pen even in the best case where a FAI takes over.

  Build­ing a pow­er­ful agent helping you get what you want, doesn’t de­stroy your abil­ity to get what you want. By my defi­ni­tion, that’s not a catas­tro­phe.

  as if a “catas­tro­phe” is nec­es­sar­ily the worst thing pos­si­ble and should be avoided at all costs. If an an­tial­igned “evil” AI were about to be re­leased with high prob­a­bil­ity, and you had a pa­per­clip max­i­mizer in a box, re­leas­ing the pa­per­clip max­i­mizer would be the best op­tion, even though that moves the chance of catas­tro­phe from high prob­a­bil­ity to in­dis­t­in­guish­able from cer­tainty.

  Cor­rect. Again, I don’t mean to say that any catas­tro­phe is liter­ally the worst out­come pos­si­ble.

The Catas­trophic Con­ver­gence Conjecture

• TurnTrout 12 Feb 2020 23:54 UTC

  3 points

  in reply to: lsusr's comment on: Chap­ter 23: Belief in Belief

  My apolo­gies for leav­ing that un­marked.

• TurnTrout 12 Feb 2020 16:01 UTC

  LW: 2 AF: 1

  AF

  in reply to: rohinmshah's comment on: At­tain­able util­ity has a sub­agent problem

  Nit­pick: “At­tain­able util­ity reg­u­lariza­tion” should be “At­tain­able util­ity preser­va­tion”

• TurnTrout 12 Feb 2020 15:48 UTC

  3 points

  in reply to: adamShimi's comment on: Value Impact

  Oh, I think I see. Yes, this is ex­plic­itly talked about later in the se­quence—“re­sources” like cash are given their im­por­tance by how they af­fect fu­ture pos­si­bil­ities, and that’s highly con­text-de­pen­dent.

  (Let me know if this still isn’t ad­dress­ing your ob­jec­tion)

• TurnTrout 12 Feb 2020 13:44 UTC

  2 points

  in reply to: adamShimi's comment on: Value Impact

  Also, I feel that some of your ex­am­ples of ob­jec­tive im­pact are in­deed rele­vant to agents in gen­eral (not dy­ing/​be­ing de­stroyed), while other de­pends on shar­ing a com­mon con­text (cash, which would be ut­terly use­less in Peb­blia if the lo­cal econ­omy was based on ex­chang­ing pee­bles for pee­bles).

  Yeah, in the post I wrote

  Even if we were on Peb­blia, we’d prob­a­bly think pri­mar­ily of the im­pact on hu­man-Peb­ble­hoarder re­la­tions.

• TurnTrout 12 Feb 2020 1:51 UTC

  21 points

  on: TurnTrout’s short­form feed

  For quite some time, I’ve dis­liked wear­ing glasses. How­ever, my eyes are sen­si­tive, so I dis­missed the pos­si­bil­ity of con­tacts.

  Over break, I re­al­ized I could still learn to use con­tacts, it would just take me longer. Sure enough, it took me an hour and five min­utes to put in my first con­tact, and I couldn’t get it out on my own. An hour of prac­tice later, I put in a con­tact on my first try, and took it out a few sec­onds later. I’m very hap­pily wear­ing con­tacts right now, as a mat­ter of fact.

  I’d suffered glasses for over fif­teen years be­cause of a cached de­ci­sion – be­cause I didn’t think to re­think some­thing liter­ally right in front of my face ev­ery sin­gle day.

  What cached de­ci­sions have you not re­con­sid­ered?