adamShimi(Adam Shimi)

• adamShimi 20 Feb 2020 22:08 UTC

  1 point

  in reply to: rohinmshah's comment on: Goal-di­rected = Model-based RL?

  Thanks for the feed­back!

  I in­deed am think­ing about your in­tu­itions for goal-di­rected be­hav­iors, be­cause it seems quite im­por­tant. I cur­rently lack a clear idea (as for­mal as pos­si­ble) of what you mean, and thus I have trou­ble weight­ing your ar­gu­ments that it is not nec­es­sary, or that it causes most prob­lems in safety. And since these ar­gu­ments would have sig­nifi­cant im­pli­ca­tions, I want to have as in­formed as pos­si­ble an opinion on them.

  Since you say that goal-di­rected be­hav­ior is not about hav­ing a model or not, is it about the form of the model? Or about the use of the model? Would a model-based agent that did not adapt its model when the en­vi­ron­ment changed be con­sid­ered as not goal-di­rected (like the lookup-table agent in your ex­am­ple)?

• adamShimi 20 Feb 2020 20:44 UTC

  2 points

  in reply to: Charlie Steiner's comment on: Goal-di­rected = Model-based RL?

  I’m cu­ri­ous about what you think peo­ple are aware of: that the idea of goal-di­rect­ed­ness from the value learn­ing se­quence is cap­tured by model-based RL, or that any suffi­ciently pow­er­ful agent (im­plic­itly goal-di­rected) needs to be model-based in­stead of model-free?

  If that’s the former, I’m re­ally in­ter­ested in links to posts and com­ments point­ing that out, as I don’t know of any. And if that’s the lat­ter, then it seems that it is goes back to ask­ing whether pow­er­ful agents must be goal-di­rected.

Goal-di­rected = Model-based RL?

• adamShimi 19 Feb 2020 12:34 UTC

  3 points

  on: Train­ing Regime Day 2: Search­ing for bugs

  For more ex­am­ples, here are the bugs I found fol­low­ing the prompts:

  • I con­sis­tently take too much time to wake up in the morn­ing, be­tween 30 min­utes and 2 hours too much.

  • When work­ing on some­thing, I tend to do “just enough” to make good progress on it, and then stop for the day. Even if I could have kept go­ing.

  • Although I am very com­fortable in con­ver­sa­tions, I have a weird anx­iety about start­ing one with a com­plete stranger.

  • I have a con­sis­tent re­luc­tance to start a new ac­tivity, like a cod­ing pro­ject or cook­ing a new recipe. Whereas I thrive on new ideas.

  • My fo­cus wanes around 1 hour af­ter I start work­ing on some­thing on the best days, and I would want more.

  • I keep pro­cras­ti­nat­ing on wash­ing my dishes.

  • I take too much time think­ing about how to do things and what I should do, and too lit­tle do­ing the things.

  • I reg­u­larly feel I’m not im­por­tant to peo­ple.

  • I have trou­ble fo­cus­ing when read­ing maths, and that’s some­thing I would want to im­prove.

  I’m not sure these are bugs at the right level, but that’s what I got out of the prompts.

• adamShimi 19 Feb 2020 12:09 UTC

  2 points

  on: Train­ing Regime Day 1: What is ap­plied ra­tio­nal­ity?

  I am not sure I un­der­stand ex­actly what you are aiming for with your take 3: is ap­plied ra­tio­nal­ity a ra­tio­nal­ity that I don’t have to fol­low when I want to use emo­tions/​in­tu­itions/​breaks, or is it a ra­tio­nal­ity that con­sid­ers these op­tions when mak­ing the de­ci­sions? The former seems to per­mis­sive, in that there is never a place where I have to use ra­tio­nal­ity, while the lat­ter might fall into the same is­sues as the restau­rant ex­am­ple, by pon­der­ing to much whether I should use in­tu­ition to choose my meal.

  That be­ing said, I like the differ­ent per­spec­tives offered by the takes.

• adamShimi 16 Feb 2020 15:03 UTC

  1 point

  on: The Rea­son­able Effec­tive­ness of Math­e­mat­ics or: AI vs sandwiches

  Nice post. Be­ing con­vinced my­self of the im­por­tance of math­e­mat­ics both for un­der­stand­ing the world in gen­eral and for the spe­cific prob­lems of AI safety, I found it in­ter­est­ing to see what ar­gu­ments you mar­shaled in and against this po­si­tion.

  About the un­rea­son­able effec­tive­ness of math­e­mat­ics, I’d like to throw the “fol­low-up” state­ment: The un­rea­son­able in­effec­tive­ness of math­e­mat­ics be­yond physics (for ex­am­ple in biol­ogy). The counter ar­gu­ment, at least for biol­ogy, is that Wigner was talk­ing a lot about differ­en­tial equa­tions, which seems some­what in­effec­tive in biol­ogy; but the­o­ret­i­cal com­puter sci­ence, which one can see as the math­e­mat­i­cal study of com­pu­ta­tion, and thus some­what a branch of math­e­mat­ics, might be bet­ter fit­ted to biol­ogy.

  A gen­eral com­ment about your per­spec­tive is that you seem to equals math­e­mat­ics with for­mal speci­fi­ca­tion and proofs. That’s not nec­es­sar­ily an is­sue, but most mod­ern math­e­mat­i­ci­ans tend to not be ex­act for­mal­ists, so I thought it im­por­tant to point out.

  For the rest of my com­ments:

  • Rather than pre­cise, I would say that math­e­mat­ics are for­mal. The differ­ence lies in the fact that a pre­cise state­ment cap­tures al­most ex­actly an idea, whereas for­mal­iza­tion provide an ob­jec­tive de­scrip­tion of… some­thing. Given that the main difficulty in ap­ply­ing math­e­mat­ics and in writ­ing speci­fi­ca­tion for for­mal meth­ods is this on­tolog­i­cal iden­ti­fi­ca­tion be­tween the for­mal­iza­tion and the ob­ject in the world, I feel that it’s a bit too easy to say that maths cap­tures the ideas pre­cisely.

  • Similarly, it is not be­cause the defi­ni­tions them­selves are un­am­bigu­ous (if they are for­mal) that their in­ter­pre­ta­tion, mean­ing and use is. I agree that a for­mal defi­ni­tion is far less am­bigu­ous than a nat­u­ral lan­guage one, but that does not mean that it is com­pletely un­am­bigu­ous. Many dis­agree­ment I had in re­search were about the in­ter­pre­ta­tion of the for­mal­isms them­selves.

  • Although I agree with the idea of math­e­mat­ics cap­tur­ing some con­cept of sim­plic­ity, I would pre­cise that it is about sim­plic­ity when all is ex­plic­ited. That’s rather ob­vi­ous for ra­tio­nal­ists. For­mal defi­ni­tions tend to be full of sub­tleties and hard to man­age, but the ex­plicit ver­sions of the “sim­pler” mod­els would ac­tu­ally be more com­plex than that.

  • Nit­pick about the “quan­ti­ta­tive”: what of ab­stract alge­bra, and all the sub­fields that are not ex­plic­itly quan­ti­ta­tive? Are they use­ful only in­so­far as they serves for the more quan­ti­ta­tive parts of maths, or am I tak­ing this ar­gu­ment too far and you just meant that one use of maths was in the quan­ti­ta­tive parts?

  • The talk about Se­rial Depth makes me think about de­con­fu­sion. I feel it is in­deed rather easy to makes some­one not con­fused about mak­ing a sand­wich, while it is still un­done for AI Safety.

  • The An­thro­pocen­trism ar­gu­ments feels right to me, but I think it doesn’t ap­ply if one is try­ing to build pro­saic al­igned AGI. Then the “most im­por­tant” is to solve rather an­thro­pocen­tric mod­els of de­ci­sion and val­ues, in­stead of ab­stract­ing them away. But I might be wrong on that one.

• adamShimi 14 Feb 2020 12:50 UTC

  6 points

  on: Dist­in­guish­ing defi­ni­tions of takeoff

  I find dis­cus­sions about AI take­off to be very con­fus­ing.

  So do I. So thanks a lot for this sum­mary!

• adamShimi 13 Feb 2020 17:09 UTC

  1 point

  on: Toy model piece #5: com­bin­ing par­tial preferences

  Why should all equiv­alence classes of linked world have the same av­er­age util­ity? That en­sures the unic­ity of the util­ity func­tion up to trans­la­tion, but I’m not sure that’s always the best way to do it. What is the in­tu­ition be­hind this spe­cific choice?

• adamShimi 12 Feb 2020 16:12 UTC

  1 point

  in reply to: TurnTrout's comment on: Value Impact

  Thanks, I’ll keep go­ing then.

• adamShimi 12 Feb 2020 13:49 UTC

  1 point

  in reply to: TurnTrout's comment on: Value Impact

  I don’t see the link with my ob­jec­tion, since you quote a part of your post when you write of value im­pact (which is de­pen­dent on the val­ues of the spe­cific agents) and I talk about the need for con­text even for ob­jec­tive im­pact (which you pre­sent as in­de­pen­dent of val­ues and ob­jec­tives of spe­cific agents)

• adamShimi 12 Feb 2020 13:02 UTC

  1 point

  on: Value Impact

  I have one po­ten­tial crit­i­cism of the ex­am­ples:

  Be­cause I was not sure what was the con­crete im­pli­ca­tion of the as­ter­oid im­pact, the re­veal was unim­pact­ful on me (pun int­eded) that it was ob­jec­tively val­ued nega­tively by any­body be­cause they risk death. Had you writ­ten that the as­ter­oid strikes near the agent, or that this causes mas­sive catas­tro­phes, then I would prob­a­bly have though that it mat­tered the same for lo­cal pee­ble­hoard­ers and for hu­mans. Also, the as­ter­oid might de­stroy peb­bles (or de­pend­ing on your defi­ni­tion of peb­ble, make new ones).

  Also, I feel that some of your ex­am­ples of ob­jec­tive im­pact are in­deed rele­vant to agents in gen­eral (not dy­ing/​be­ing de­stroyed), while other de­pends on shar­ing a com­mon con­text (cash, which would be ut­terly use­less in Peb­blia if the lo­cal econ­omy was based on ex­chang­ing pee­bles for pee­bles).

  Do you just always con­sider this con­text as im­plicit?

• adamShimi 12 Feb 2020 12:33 UTC

  5 points

  in reply to: Stuart_Armstrong's comment on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  Thanks, I’m look­ing into the toy model. :)

• adamShimi 11 Feb 2020 17:19 UTC

  1 point

  on: Toy model piece #4: par­tial prefer­ences, re-re-visited

  I re­ally like the re­fine­ment of the for­mal­iza­tion, with the ex­pla­na­tions of what to keep and what was miss­ing.

  That said, I feel like the fi­nal for­mal­iza­tion could be defined di­rectly as a spe­cial type of pre­order, one com­posed only of dis­joint chains and cy­cles. Be­cause as I un­der­stand the rest of the post, that is what you use when com­put­ing the util­ity func­tion. This for­mal­iza­tion would also be more di­rect, with one less layer of ab­strac­tion.

  Is there any rea­son to pre­fer the “in­jec­tive func­tion” defi­ni­tion to the “spe­cial pre­order” one?

• adamShimi 11 Feb 2020 13:02 UTC

  5 points

  on: The Re­la­tional Stance

  Another modal­ity of re­lat­ing in­tro­duced to me by a friend a cou­ple of weeks ago is “what kind of ex­pe­rience do you take from this re­la­tion”. My friend has a quite idiosyn­cratic clas­sifi­ca­tion, but you could sep­a­rate peo­ple you see be­tween com­bi­na­tions of in­tel­lec­tual stim­u­la­tion, sense of se­cu­rity, be­ing cared for… In my mind this is quite or­thog­o­nal to other di­rec­tions: what­ever this re­la­tion holds for you, it might mat­ter tremen­dously or very lit­tle.

  The main use I have for this modal­ity is to clar­ify what I am miss­ing in my life. For ex­am­ple, when I feel lonely, I feel a dis­crep­ancy with my so­cial situ­a­tion: I have many friends, some re­ally close who care about me and about whom I care. But when con­sid­er­ing what ex­pe­rience I feel I am miss­ing in my re­la­tion­ships, I can say that it’s at­trac­tion and pas­sion for the other and sex­ual ten­sion and ac­tion.

• adamShimi 11 Feb 2020 12:45 UTC

  1 point

  in reply to: pjeby's comment on: The Curse Of The Counterfactual

  Yes, I agree that you are fo­cus­ing more on how to see the mis­take in a meta-way, in­stead of an out­side view as Nate do.

  Though I don’t think your ex­am­ple of the dis­tinc­tion is ex­actly the right one: the idea from Nate of ban­ning “should” or cash­ing out “should” would be able IMHO to un­earth the un­der­ly­ing “I should be tak­ing things se­ri­ously” ap­ply the con­se­quen­tial­ist anal­y­sis of “you will not be mea­sured by how you felt or who you pun­ished. You will be mea­sured by what ac­tu­ally hap­pened, as will we all” (para­phras­ing). What I feel is differ­ent is that the Way provide a mean for sys­tem­at­i­cally find­ind this un­der­ly­ing should and ex­plain­ing it from the in­side.

  Nonethe­less, I find both use­ful, and I am bet­ter for hav­ing the Curse of the Coun­ter­fac­tual in my men­tal toolbox.

• adamShimi 10 Feb 2020 13:05 UTC

  4 points

  in reply to: adamShimi's comment on: Cat­e­gory The­ory Without The Baggage

  I just found an­other in­ter­est­ing refer­ence: Cat­e­gories for the prac­tis­ing physi­cist. Although this is not ex­actly about dis­card­ing un­due ab­strac­tion, it does pre­sent many con­cepts in terms of con­crete ex­am­ples, and there are even real-world cat­e­gories defined in it!

• adamShimi 10 Feb 2020 12:19 UTC

  6 points

  on: The Curse Of The Counterfactual

  Great post! I want to chew on it a bit be­fore mak­ing a longer com­ment, but I no­ticed similar­i­ties be­tween this post and Nate Soares’s Re­plac­ing Guilt se­quence (which I con­sider the most im­por­tant se­quence… ever). More speci­fi­cally, he seems to say things similar about guilt and should in “should” con­sid­ered harm­ful, Not be­cause you “should” and Your “shoulds” are not a duty.

  For ex­am­ple, from “should” con­sid­ered harm­ful:

  I see lots of guilt-mo­ti­vated peo­ple use “shoulds” as ul­ti­ma­tums: “ei­ther I get the meds, or I am a bad per­son.” They leave them­selves only two choices: go out of their way on the way to work and suffer through awk­ward hu­man in­ter­ac­tion at the phar­macy, or be bad. Either way, they lose: the should has set them up for failure.

  But the ac­tual op­tions aren’t “suffer” or “be bad.” The ac­tual op­tions are “in­cur the so­cial/​time costs of buy­ing meds” or “in­cur the phys­i­cal/​men­tal costs of feel­ing ill.” It’s just a choice: you weigh the branches, and then you pick. Nei­ther branch makes you “bad.” It’s ok to de­cide that the so­cial/​time costs out­weigh the phys­i­cal/​men­tal costs. It’s ok to de­cide the op­po­site. Nei­ther side is a “should.” Both sides are an op­tion.

  Or the idea of prefer­ing to pun­ish some­one (me or an­other) in­stead of ac­tu­ally look­ing at the situ­a­tion and ac­cept­ing it, makes me think of tol­er­ifi­ca­tion:

  There’s a cer­tain type of dark­ness in the world that most peo­ple sim­ply can­not to see. It’s not the ab­stract dark­ness: peo­ple will read­ily ac­knowl­edge that the world is bro­ken, and ex­plain how and why the hated out-group is re­spon­si­ble. And that’s ex­actly what I’m point­ing at: upon see­ing that the world is bro­ken, peo­ple ex­pe­rience an im­pulse to ex­plain the bro­ken­ness in a way that re­lieves the ten­sion. When see­ing that the world is bro­ken, peo­ple re­flex­ively feel a need to ex­plain. Carol can ac­knowl­edge that there is suffer­ing abroad, but this ac­knowl­edge­ment comes part and par­cel with an ex­pla­na­tion about why she bears no re­spon­si­bil­ity. Dave can ac­knowl­edge that he failed to pass the in­ter­view, but his mind au­to­mat­i­cally gen­er­ates rea­sons why this is an ac­cept­able state of af­fairs.

  This is the type of dark­ness in the world that most peo­ple can­not see: they can­not see a world that is un­ac­cept­able. Upon notic­ing that the world is bro­ken, they re­flex­ively list rea­sons why it is still tol­er­able. Even cyn­i­cism, I think, can fill this role: I of­ten read cyn­i­cism as an at­tempt to ex­plain a world full of cal­lous ne­glect and ca­sual cru­elty, in a frame­work that makes ne­glect and cru­elty seem nat­u­ral and ex­pected (and there­fore tol­er­able).

  I call this re­flex­ive re­sponse “tol­er­ifi­ca­tion,” and if you watch for it, you can see it ev­ery­where.

  The ap­proach of these ques­tions in the re­plac­ing guilt se­ries is not ex­actly at the same level; most no­tably, I feel Nate is try­ing to ex­plain why should are not “use­ful” and cause only harm that can­not serve for ac­com­plish­ing your goals. On the other hand, I see this post as more about ex­am­in­ing the ex­act mechanism un­der­ly­ing this er­ror we make.

  Still, I feel the con­nec­tion is strong enough to en­courage peo­ple to read both.

• adamShimi 10 Feb 2020 11:26 UTC

  3 points

  on: What can the prin­ci­pal-agent liter­a­ture tell us about AI risk?

  Great post! It ex­plained clearly both po­si­tions, clar­ified the po­ten­tial uses of PAL and pro­posed vari­a­tions when it was con­sid­ered ac­cessible.

  Maybe my only is­sue is with the (lack of) defi­ni­tion of the prin­ci­pal-agent prob­lem. The rest of the post works rel­a­tively well with­out you defin­ing it ex­plic­itly, but I think a short defi­ni­tion (even just a rephras­ing of the one on Wikipe­dia) would make the post even more read­able.

• adamShimi 9 Feb 2020 15:22 UTC

  11 points

  in reply to: johnswentworth's comment on: What Money Can­not Buy

  Okay, so we agree that it’s im­prob­a­ble (at least for de­ci­sion prob­lems) to be able to ver­ify an an­swer faster than find­ing it. What you care about are cases where ver­ifi­ca­tion is eas­ier, as is con­jec­tured for ex­am­ple for NP (where ver­ifi­ca­tion is polyno­mial, but find­ing an an­swer is sup­posed to not be).

  For IP, if we only want to ver­ify any real-world prop­erty, I ac­tu­ally have a sim­ple ex­am­ple I give into my in­tro to com­plex­ity the­ory lec­tures. Imag­ine that you are color-blind (pre­cisely, a spe­cific red and a spe­cific green look ex­actly the same to you). If I have two balls, perfectly similar ex­cept one is green and the other is red, I can con­vince you that these balls are of differ­ent col­ors. It is ba­si­cally the in­ter­ac­tive pro­to­col for graph non-iso­mor­phism: you flip a coin, and de­pend­ing on the re­sult, you ex­change the balls with­out me see­ing it. If I can tell whether you ex­changed the balls a suffi­cient num­ber of times, then you should get con­vinced that I can ac­tu­ally differ­en­ti­ate them.

  Of course this is not nec­es­sar­ily ap­pli­ca­ble to ques­tions like tastes. More­over, it is a pro­to­col for show­ing that I can dis­t­in­guish be­tween the balls; it does not show why.

• adamShimi 9 Feb 2020 13:37 UTC

  1 point

  on: Re­search Agenda v0.9: Syn­the­sis­ing a hu­man’s prefer­ences into a util­ity function

  Could you give a list of some open prob­lems or open ques­tions re­lated to this agenda (maybe with some poin­t­ers to the more rele­vant posts)? I am po­ten­tially in­ter­ested in work­ing on it, but I find it far eas­ier to study a topic (and you sir write a lot of tech­ni­cal posts) while try­ing to solve some con­crete prob­lem.

  Thanks in ad­vance!