TurnTrout

• TurnTrout 19 Sep 2019 16:04 UTC

  2 points

  in reply to: TurnTrout's comment on: TurnTrout’s short­form feed

  Sure, if peo­ple are asked to solve a prob­lem and say they can’t af­ter two sec­onds, yes—make fun of that a bit. But that two sec­onds cov­ers more ground than you might think, due to Sys­tem 1 pre­com­pu­ta­tion.

  Ex­pand­ing on this, there is an as­pect of Ac­tu­ally Try­ing that is prob­a­bly miss­ing from S1 pre­com­pu­ta­tion. So, maybe the two-sec­ond “at­tempt” is ac­tu­ally use­less for most peo­ple be­cause sub­con­scious de­liber­a­tion isn’t hardass enough at giv­ing its all, at mak­ing des­per­ate and ex­traor­di­nary efforts to solve the prob­lem.

• TurnTrout 18 Sep 2019 21:57 UTC

  6 points

  on: TurnTrout’s short­form feed

  Good, origi­nal think­ing feels pre­sent to me—as if men­tal re­sources are well-al­lo­cated.

  The thought which prompted this:

  Sure, if peo­ple are asked to solve a prob­lem and say they can’t af­ter two sec­onds, yes—make fun of that a bit. But that two sec­onds cov­ers more ground than you might think, due to Sys­tem 1 pre­com­pu­ta­tion.

  Re­act­ing to a bit of HPMOR here, I no­ticed some­thing felt off about Harry’s re­ply to the Fred/​Ge­orge-tried-for-two-sec­onds thing. Hav­ing a bit of ex­pe­rience notic­ing con­fus­ing, I did not think “I no­tice I am con­fused” (al­though this can be use­ful). I did not think “Eliezer prob­a­bly put thought into this”, or “Harry is kinda dumb in cer­tain ways—so what if he’s a bit un­fair here?”. Without re­sur­fac­ing, or dis­trac­tion, or won­der­ing if this train of thought is more fun than just read­ing fur­ther, I just thought about the ob­ject-level ex­change.

  Peo­ple need to al­lo­cate men­tal en­ergy wisely; this goes far be­yond fo­cus­ing on im­por­tant tasks. Your ex­ist­ing men­tal skil­lsets already op­ti­mize and auto-pi­lot cer­tain men­tal mo­tions for you, so you should al­lo­cate less de­liber­a­tion to them. In this case, the con­fu­sion-notic­ing mod­ule was honed; by not wor­ry­ing about how well I no­ticed con­fu­sion, I was able to quickly have an origi­nal thought.

  When thought pro­cesses de­rail or brain­storm­ing ses­sions bear no fruit, in­ap­pro­pri­ate al­lo­ca­tion may be to blame. For ex­am­ple, if you’re anx­ious, you’re in­ter­rupt­ing the ac­tual thoughts with “what-if”s.

  To con­trast, non-pre­sent think­ing feels like a con­trol­ler di­rect­ing thoughts to go from here to there: do this and then, check that, come up for air over and over… Pre­sent think­ing is a stream of un­in­ter­rupted strikes, the train of thought chug­ging along with­out self-con­scious­ness. Mov­ing, in­stead of think­ing about mov­ing while mov­ing.

  I don’t know if I’ve nailed down the thing I’m try­ing to point at yet.

• TurnTrout 17 Sep 2019 17:24 UTC

  2 points

  on: Chap­ter 23: Belief in Belief

  I don’t un­der­stand why Time set­tles upon this re­s­olu­tion at the end. Why is the world where Flitwick comes to get him one of the sim­plest con­sis­tent ones (which is later heav­ily im­plied to be the crite­rion)?

• TurnTrout 16 Sep 2019 23:16 UTC

  3 points

  in reply to: elriggs's comment on: What You See Isn’t Always What You Want

  I was think­ing of it be­ing over ob­ser­va­tions, but hav­ing it be over States x Ac­tions leads to a po­ten­tially differ­ent out­come. An $n$-step ver­sion is, your re­ward func­tion is a map­ping $O^n\mapsto R$ (you’re grad­ing the last $n$ ob­ser­va­tions jointly). Eg in Atari DRL you might see the last four frames be­ing fed to the agent as an ap­prox­i­ma­tion (since the games might well be 4-step Marko­vian; that is, the four pre­vi­ous time steps fully de­ter­mine what hap­pens next).

• TurnTrout 16 Sep 2019 22:19 UTC

  6 points

  on: TurnTrout’s short­form feed

  I seem to differ­ently dis­count differ­ent parts of what I want. For ex­am­ple, I’m some­what will­ing to post­pone fun to low-prob­a­bil­ity high-fun fu­tures, whereas I’m not will­ing to do the same with ro­mance.

• TurnTrout 15 Sep 2019 6:39 UTC

  LW: 3 AF: 1

  AF

  in reply to: FactorialCode's comment on: What You See Isn’t Always What You Want

  They’re definitely not suffi­cient, al­most cer­tainly. A full fledged di­a­mond max­i­mizer would need far more ma­chin­ery, if only to do the max­i­miza­tion and prop­erly learn the rep­re­sen­ta­tion.

  Clar­ifi­ca­tion: I meant (but in­ad­e­quately ex­pressed) “do you think any rea­son­able ex­ten­sion of these kinds of ideas could get what we want?” Ob­vi­ously, it would be a quite un­fair de­mand for rigor to de­mand whether we can do the thing right now.

  Thanks for the great re­ply. I think the re­main­ing dis­agree­ment might boil down to the ex­pected difficulty of avoid­ing Good­hart here. I do agree that us­ing rep­re­sen­ta­tions is a way around this is­sue, and it isn’t the rep­re­sen­ta­tion learn­ing ap­proach’s job to si­mul­ta­neously deal with Good­hart­ing.

• TurnTrout 14 Sep 2019 16:32 UTC

  2 points

  in reply to: jmh's comment on: The Power to Un­der­stand “God”

  I am not defend­ing or re­fut­ing any­thing here but will point out that athe­ism is a state­ment about some­thing not ex­ist­ing. Prov­ing some­thing does not ex­ist is a highly prob­le­matic ex­er­cise.

  This feels like a fram­ing prob­lem. For me to think that my room isn’t full of ap­ples, I don’t need to prove it isn’t full of ap­ples, I just need to as­sign sig­nifi­cantly higher prob­a­bil­ity to ap­ples not burst­ing out of my win­dows.

• TurnTrout 14 Sep 2019 16:30 UTC

  3 points

  in reply to: shminux's comment on: The Power to Un­der­stand “God”

  Uh. Depends on how you define be­ing ra­tio­nal. If you fol­low Eliezer and define it as win­ning, then there are many be­liev­ers that are way ahead of you.

  If you aren’t con­trol­ling for con­found­ing fac­tors, like be­ing born into an ex­tremely rich fam­ily, and in­stead just com­pare the most suc­cess­ful be­liev­ers and the most suc­cess­ful ra­tio­nal­ists (or, in this case, Liron in par­tic­u­lar), of course we’re go­ing to get blown out of the wa­ter. There are how many ra­tio­nal­ists, again? The in­ter­est­ing thing is to ask, if we con­trol for all rele­vant fac­tors, does ra­tio­nal­ity train­ing have a good effect size? This is a good ques­tion, with quite a bit of pre­vi­ous dis­cus­sion.

  If you’ll al­low me to guess one po­ten­tial re­sponse, let’s sup­pose there’s no effect. What then—are we all be­ing “ir­ra­tional”, and is the en­tire ra­tio­nal­ity pro­ject a failure? Not nec­es­sar­ily. This de­pends on what progress is be­ing made (as benefits can be non­lin­ear in skill level). For ex­am­ple, maybe I’m learn­ing Shaolin Kenpo, and I go from white to yel­low belt. I go up to a buff guy on the street and get my ass kicked. Have I failed to learn any Shaolin Kenpo?

• TurnTrout 14 Sep 2019 7:28 UTC

  LW: 3 AF: 1

  AF

  in reply to: FactorialCode's comment on: What You See Isn’t Always What You Want

  I wasn’t aware of that work, thanks for link­ing! It’s true that we don’t have to spec­ify the rep­re­sen­ta­tion; in­stead, we can learn it. Do you think we could build a di­a­mond max­i­mizer us­ing those ideas, though? The con­cern here is that the rep­re­sen­ta­tion has to cleanly de­mar­cate what we think of as di­a­monds, if we want the op­ti­mal policy to en­tail ac­tu­ally max­i­miz­ing di­a­monds in the real world. This prob­lem tastes like it has a bit of that ‘fun­da­men­tally in­tractable’ fla­vor.

• TurnTrout 13 Sep 2019 17:33 UTC

  LW: 3 AF: 1

  AF

  in reply to: FactorialCode's comment on: What You See Isn’t Always What You Want

  I feel some­what pes­simistic about do­ing this ro­bustly enough to scale to AGI. From an ear­lier com­ment of mine:

  It isn’t ob­vi­ous to me that spec­i­fy­ing the on­tol­ogy is sig­nifi­cantly eas­ier than spec­i­fy­ing the right ob­jec­tive. I have an in­tu­ition that on­tolog­i­cal ap­proaches are doomed. As a sim­ple case, I’m not aware of any fun­da­men­tal progress on build­ing some­thing that ac­tu­ally max­i­mizes the num­ber of di­a­monds in the phys­i­cal uni­verse, nor do I think that such a thing has a nat­u­ral, sim­ple de­scrip­tion.

What You See Isn’t Always What You Want

• TurnTrout 11 Sep 2019 16:30 UTC

  LW: 5 AF: 2

  AF

  on: Coun­ter­fac­tual Or­a­cles = on­line su­per­vised learn­ing with ran­dom se­lec­tion of train­ing episodes

  if the only thing that ever sees the Or­a­cle’s out­put dur­ing a train­ing epi­sode is an au­to­mated sys­tem that com­putes the Or­a­cle’s re­ward/​loss, and that sys­tem is se­cure be­cause it’s just com­put­ing a sim­ple dis­tance met­ric (com­par­ing the Or­a­cle’s out­put to the train­ing la­bel), then re­ward hack­ing and self-con­firm­ing pre­dic­tions can’t hap­pen.

  This as­sump­tion seems large, and I don’t un­der­stand why we can make it. The loss is com­puted against the out­put of a phys­i­cal sys­tem in the real world which is not, in fact, causally in­de­pen­dent of the or­a­cle’s pre­dic­tions. It is not Pla­ton­i­cally graded on how cor­rect it is with re­spect to what we had in mind. If the pro­posal is “just make the grad­ing sys­tem re­ally hard to hack”, that runs into the same prob­lems as box­ing—you’re still run­ning a sys­tem that is look­ing for ways to hurt you.

• TurnTrout 4 Sep 2019 15:52 UTC

  4 points

  in reply to: sayan's comment on: Towards a New Im­pact Measure

  Read the pa­per for now, and read the up­com­ing Refram­ing Im­pact se­quence later this year. There is a sig­nifi­cant differ­ence, but this post seems bad at com­mu­ni­cat­ing the key paradigm shifts I origi­nally en­vi­sioned com­mu­ni­cat­ing (hence the se­quence).

• TurnTrout 29 Aug 2019 14:55 UTC

  LW: 4 AF: 2

  AF

  in reply to: Vika's comment on: Rev­ersible changes: con­sider a bucket of water

  I agree that it’s not the core, and I think this is a very co­gent sum­mary. There’s a deeper dis­agree­ment about what we need done that I’ll lay out in de­tail in Refram­ing Im­pact.

• TurnTrout 27 Aug 2019 14:54 UTC

  6 points

  in reply to: BrandonReinhart's comment on: The Apol­o­gist and the Revolutionary

  It may be that nor­mal brains would ex­pe­rience no change in the func­tion of ra­tio­nal­ity com­pared to ar­bi­trar­ily dam­aged brains. The effects of vestibu­lar stim­u­la­tion are so varied that us­ing it to af­fect some spe­cific re­sult sounds like the neu­ro­scien­tific equiv­a­lent to hit­ting the TV on the side to clear static.

  That’s too bad; I was think­ing it might be a nice way to kick-start a good ole Cri­sis of Faith.

  ETA: And so did a mil­lion other peo­ple down­thread.

• TurnTrout 27 Aug 2019 1:32 UTC

  LW: 2 AF: 1

  AF

  in reply to: Stuart_Armstrong's comment on: Rev­ersible changes: con­sider a bucket of water

  The agent would still walk around for rea­son­able far­sight­ed­ness; kick­ing the bucket into the pool per­turbs most AUs. There’s no real “risk” to not kick­ing the bucket.

  AUP is only defined over states for MDPs be­cause states are the ob­ser­va­tions. AUP in par­tially ob­serv­able en­vi­ron­ments uses re­ward func­tions over sen­sory in­puts. Again, I as­sert we don’t need to think about molecules or on­tolo­gies.

  But, as cov­ered in the dis­cus­sion linked above, wor­ry­ing about pe­nal­iz­ing molec­u­lar shifts or not misses the point of im­pact: the agent doesn’t catas­troph­i­cally reap­pro­pri­ate our power, so we can still get it to do what we want. (The thread is an­other place to re­cal­ibrate on what I’ve tried to com­mu­ni­cate thus far.)

  The truth is that, with a tiny set of ex­cep­tions, all our ac­tions are ir­re­versible, shut­ting down many pos­si­bil­ities for ever.

  AUP doesn’t care much at all about literal re­versibil­ity.

• TurnTrout 26 Aug 2019 23:00 UTC

  LW: 7 AF: 4

  AF

  on: Rev­ersible changes: con­sider a bucket of water

  I think this is the wrong fram­ing, rest­ing on un­nec­es­sary as­sump­tions about how im­pact must be solved. You’re con­cerned that it’s hard to look at the world and in­fer what it means for peo­ple to be “min­i­mally im­pacted”. I agree: that’s re­ally hard. I get the con­cern. We shouldn’t do that, and that’s not what AUP aims to do.

  When I say “we shouldn’t do that, and that’s not what AUP aims to do”, I re­ally, re­ally, re­ally mean it. (IIRC) I’ve never pro­posed do­ing it that way (al­though to be fair, my ini­tial AUP post didn’t re­peat this point enough). If e.g. my pre­sen­ta­tion doesn’t com­mu­ni­cate the bet­ter frame, wait for the “Refram­ing Im­pact” se­quence!

• TurnTrout 26 Aug 2019 19:53 UTC

  2 points

  on: Schel­ling Cat­e­gories, and Sim­ple Mem­ber­ship Tests

  why wouldnt you just use x_41 all of the time if you only get one cat­e­gory—why is this differ­ent from what you’d pick at a gods-eye view?

• TurnTrout 23 Aug 2019 19:57 UTC

  9 points

  on: Ac­tu­ally updating

  To make this clearer, imag­ine some­thing hap­pens (event Y) that changes your opinion of X. Maybe you re­al­ise that the thing ‘shouldn’t’ be much ev­i­dence at all. You de­cide it will only slightly af­fect your opinion. You may end up with a cer­tain con­clu­sion by look­ing at old ev­i­dence and this thing Y (that you have ac­tively de­cided to only weight a small amount). Then, some­thing may then hap­pen that makes you re­ally be­lieve that thing Y is small ev­i­dence. Sud­denly, you are look­ing at the same ev­i­dence and this thing Y (weighted the same amount as you tried to weight it be­fore), but you have a differ­ent con­clu­sion. It’s like you are look­ing at the ev­i­dence through a differ­ent ‘lens’. It seems like you are able to ex­ploit un­cer­tainty around ev­i­dence to get differ­ing con­clu­sions de­pend­ing on this ‘lens’. This whole thing might make you feel large in­ter­nal dis­trust.

  Can you give an illus­tra­tive ex­am­ple?

• TurnTrout 23 Aug 2019 17:55 UTC

  LW: 4 AF: 3

  AF

  on: When do util­ity func­tions con­strain?

  Th­ese thoughts feel im­por­tant:

  our ac­tions will change the amount of util­ity we ex­pect to be available. This is not be­cause of the class of po­ten­tial util­ity func­tions ex­actly, but be­cause the ac­tion space/​util­ity func­tion class com­bi­na­tion are such that the ac­tions and changes in mag­ni­tude of available util­ity are always linked

  What we need to find, for a given agent to be con­strained by be­ing a ‘util­ity max­imiser’ is to con­sider it as hav­ing a mem­ber of a class of util­ity func­tions where the ac­tions that are available to it sys­tem­at­i­cally al­ter the ex­pected util­ity available to it—for all util­ity func­tions within this class. This is a nec­es­sary con­di­tion for util­ity func­tions to re­strict be­havi­our, not a suffi­cient one.

  It turns out that available util­ity (canon­i­cally, at­tain­able util­ity, or AU) tracks with other im­por­tant ques­tions of when and why we can con­strain our be­liefs about an agent’s ac­tions. See, shift­ing from think­ing about util­ity to abil­ity to get util­ity lets us for­mally un­der­stand in­stru­men­tal con­ver­gence (se­quence up­com­ing, so no cita­tion yet). Eg us­ing MDPs as the ab­strac­tion, the thing hap­pen­ing when AU changes also causes in­stru­men­tal con­ver­gence to arise from the struc­tural prop­er­ties of the en­vi­ron­ment. I think this holds for quite a few dis­tri­bu­tions over re­ward func­tions, in­clud­ing at least the uniform dis­tri­bu­tion. So, it feels like you’re onto some­thing with the re­stric­tion you point at.

  Note that within al­most any nat­u­ral class there will be the de­gen­er­ate util­ity func­tion in which all re­sults re­sult in equal util­ity and there­fore all ac­tions are per­mis­si­ble—this must be de­liber­ately ex­cluded to make pre­dic­tions.

  The note seems un­nec­es­sary (if I read cor­rectly), as the AU doesn’t change for those util­ity func­tions?