abramdemski

Align­ment Re­search Field Guide

Pavlov Generalizes

• abramdemski 22 Jan 2019 1:47 UTC

  4 points

  AF

  in reply to: rohinmshah's comment on: Align­ment Newslet­ter #41

  This seems like it is not about the “mo­ti­va­tional sys­tem”, and if this were im­ple­mented in a robot that does have a sep­a­rate “mo­ti­va­tional sys­tem” (i.e. it is goal-di­rected), I worry about a near­est un­blocked strat­egy.

  I am con­fused about where you think the mo­ti­va­tion sys­tem comes into my state­ment. It sounds like you are imag­in­ing that what I said is a con­straint, which could some­how be cou­pled with a seper­ate mo­ti­va­tion sys­tem. If that’s your in­ter­pre­ta­tion, that’s not what I meant at all, un­less ran­dom sam­pling counts as a mo­ti­va­tion sys­tem. I’m say­ing that all you do is sam­ple from what’s con­sented to.

  But, maybe what you are say­ing is that in “the in­ter­sec­tion of what the user ex­pects and what the user wants”, the first is func­tion­ing as a con­straint, and the sec­ond is func­tion­ing as a mo­ti­va­tion sys­tem (ba­si­cally the usual IRL mo­ti­va­tion sys­tem). If that’s what you meant, I think that’s a valid con­cern. What I was imag­in­ing is that you are try­ing to in­fer “what the user wants” not in terms of end goals, but rather in terms of ac­tions (re­ally, poli­cies) for the AI. So, it is more like an ap­proval-di­rected agent to an ex­tent. If the hu­man says “get me gro­ceries”, the job of the AI is not to in­fer the end state the hu­man is ask­ing the robot to op­ti­mize for, but rather, to in­fer the set of poli­cies which the hu­man is try­ing to point at.

  There’s no op­ti­miza­tion on top of this find­ing per­verse in­stan­ti­a­tions of the con­straints; the AI just fol­lows the policy which it in­fers the hu­man would like. Of course the pow­er­ful learn­ing sys­tem re­quired for this to work may per­versely in­stan­ti­ate these be­liefs (ie, there may be dae­mons aka in­ner op­ti­miz­ers).

  (The most ob­vi­ous prob­lem I see with this ap­proach is that it seems to im­ply that the AI can’t help the hu­man do any­thing which the hu­man doesn’t already know how to do. For ex­am­ple, if you don’t know how to get started filing your taxes, then the robot can’t help you. But maybe there’s some way to differ­en­ti­ate be­tween more be­nign cases like that and less be­nign cases like us­ing nan­otech­nol­ogy to more effec­tively get gro­ceries?)

  A third in­ter­pre­ta­tion of your con­cern is that you’re say­ing that if the thing is do­ing well enough to get gro­ceries, there has to be pow­er­ful op­ti­miza­tion some­where, and wher­ever it is, it’s go­ing to be push­ing to­ward per­verse in­stan­ti­a­tions one way or an­other. I don’t have any ar­gu­ment against this con­cern, but I think it mostly amounts to a con­cern about in­ner op­ti­miz­ers.

  (I feel com­pel­led to men­tion again that I don’t feel strongly that the whole idea makes any sense. I just want to con­vey why I don’t think it’s about con­strain­ing an un­der­ly­ing mo­ti­va­tion sys­tem.)

• abramdemski 20 Jan 2019 7:59 UTC

  8 points

  AF

  on: Align­ment Newslet­ter #41

  Non-Con­se­quen­tial­ist Co­op­er­a­tion? (Abram Dem­ski): [...]

  How­ever, this also feels differ­ent from cor­rigi­bil­ity, in that it feels more like a limi­ta­tion put on the AI sys­tem, while cor­rigi­bil­ity seems more like a prop­erty of the AI’s “mo­ti­va­tional sys­tem”. This might be fine, since the AI might just not be goal-di­rected. One other benefit of cor­rigi­bil­ity is that if you are “some­what” cor­rigible, then you would like to be­come more cor­rigible, since that is what the hu­man would pre­fer; in­formed-con­sent-AI doesn’t seem to have an analo­gous benefit.

  You could definitely think of it as a limi­ta­tion to put on a sys­tem, but I ac­tu­ally wasn’t think­ing of it that way when I wrote the post. I was try­ing to imag­ine some­thing which only op­er­ates from this prin­ci­ple. Granted, I didn’t re­ally ex­plain how that could work. I was imag­in­ing that it does some­thing like sam­ple from a prob­a­bil­ity dis­tri­bu­tion which is (speak­ing in­tu­itively) the in­ter­sec­tion of what you ex­pect it to do and what you would like it to do.

  (It now seems to me that al­though I put “non-con­se­quen­tial­ist” in the ti­tle of the post, I didn’t ex­plain the part where it isn’t con­se­quen­tial­ist very well. Which is fine, since the post was very much just spit­bal­ling.)

• abramdemski 19 Jan 2019 20:22 UTC

  2 points

  AF

  in reply to: Wei_Dai's comment on: CDT=EDT=UDT

  Agreed. I’ll at least edit the post to point to this com­ment.

• abramdemski 18 Jan 2019 21:41 UTC

  2 points

  AF

  in reply to: Vaniver's comment on: CDT=EDT=UDT

  I’m not sure which you’re ad­dress­ing, but, note that I’m not ob­ject­ing to the prac­tice of illus­trat­ing vari­ables with di­a­monds and boxes rather than only cir­cles so that you can see at a glance where the choices and the util­ity are (al­though I don’t tend to use the con­ven­tion my­self). I’m ob­ject­ing to the fur­ther im­pli­ca­tion that do­ing this makes it not a Bayes net.

• abramdemski 18 Jan 2019 18:57 UTC

  3 points

  AF

  in reply to: Daniel Kokotajlo's comment on: XOR Black­mail & Causality

  I hear there is a way to fid­dle with the foun­da­tions of prob­a­bil­ity the­ory so that con­di­tional prob­a­bil­ities are taken as ba­sic and or­di­nary prob­a­bil­ities are defined in terms of them. Maybe this would solve the prob­lem?

  This does help some­what. See here. But, in or­der to get good an­swers from that, you need to already know enough about the struc­ture of the situ­a­tion.

  Maybe I’m late to the party, in which case sorry about that & I look for­ward to hear­ing why I’m wrong, but I’m not con­vinced that ep­silon-ex­plo­ra­tion is a satis­fac­tory way to en­sure that con­di­tional prob­a­bil­ities are well-defined. Here’s why:

  I agree, but I also think there are some things point­ing in the di­rec­tion of “there’s some­thing in­ter­est­ing go­ing on with ep­silon ex­plo­ra­tion”. Speci­fi­cally, there’s a pretty strong anal­ogy be­tween ep­silon ex­plo­ra­tion and modal UDT: MUDT is like the limit as you send ex­plo­ra­tion prob­a­bil­ity to zero, so it never ac­tu­ally hap­pens but it still hap­pens in non­stan­dard mod­els. How­ever, that only seems to work when you know the struc­ture of the situ­a­tion log­i­cally. When you have to learn it, you have to ac­tu­ally ex­plore some­times to get it right.

  To the ex­tent that MUDT looks like a deep re­sult about coun­ter­fac­tual rea­son­ing, I take this as a point in fa­vor of ep­silon ex­plo­ra­tion tel­ling us some­thing about the deep struc­ture of coun­ter­fac­tual rea­son­ing.

  Any­way, see here for some more re­cent thoughts of mine. (But I didn’t dis­cuss the ques­tion of ep­silon ex­plo­ra­tion as much as I could have.)

• abramdemski 18 Jan 2019 18:40 UTC

  2 points

  AF

  in reply to: Vaniver's comment on: CDT=EDT=UDT

  I dis­agree. All the nodes in the net­work should be thought of as ground­ing out in imag­i­na­tion, in that it’s a world-model, not a world. Maybe I’m not see­ing your point.

  I would definitely like to see a graph­i­cal model that’s more ca­pa­ble of rep­re­sent­ing the way the world-model it­self is re­cur­sively in­volved in de­ci­sion-mak­ing.

  One ar­gu­ment for call­ing an in­fluence di­a­gram a gen­er­al­iza­tion of a bayes could be that the con­di­tional prob­a­bil­ity table for the agent’s policy given ob­ser­va­tions is not given as part of the in­fluence di­a­gram, and in­stead must be solved for. But we can still think of this as a spe­cial case of a Bayes net, rather than a gen­er­al­iza­tion, by think­ing of an in­fluence di­a­gram as a spe­cial sort of Bayes net in which the de­ci­sion nodes have to have con­di­tional prob­a­bil­ity ta­bles obey­ing some op­ti­mal­ity no­tion (such as the CDT op­ti­mal­ity no­tion, the EDT op­ti­mal­ity no­tion, etc).

  This con­straint is not eas­ily rep­re­sented within the Bayes net it­self, but in­stead im­posed from out­side. It would be nice to have a graph­i­cal model in which you could rep­re­sent that kind of con­straint nat­u­rally. But sim­ply la­bel­ling things as de­ci­sion nodes doesn’t do much. I would rather have a way of iden­ti­fy­ing some­thing as agent-like based on the struc­ture of the model for it. (To give a re­ally bad ver­sion: sup­pose you al­low di­rected cy­cles, rather than re­quiring DAGs, and you think of the “back­wards causal­ity” as agency. But, this is re­ally bad, and I offer it only to illus­trate the kind of thing I mean—al­low­ing you to ex­press the struc­ture which gives rise to agency, rather than tak­ing agency as a new prim­i­tive.)

• abramdemski 18 Jan 2019 18:28 UTC

  2 points

  in reply to: Quinn's comment on: What makes peo­ple in­tel­lec­tu­ally ac­tive?

  All in all, I can’t wrap my head around “what is the differ­ence be­tween a pro­ducer and a con­sumer of thought?” be­cause the ques­tion as posed seems to hold rigor, even qual­ity, con­stant/​ir­rele­vant.

  I’m not try­ing to hold it con­stant, I’m just try­ing to un­der­stand a rel­a­tively low stan­dard, be­cause that’s the part I feel con­fused about. It seems rel­a­tively much eas­ier to look at bad in­tel­lec­tual out­put and say how it could have been bet­ter, think about the thought pro­cesses in­volved, etc. Much harder to say what goes into pro­duc­ing out­put at all vs not do­ing so.

• abramdemski 16 Jan 2019 18:26 UTC

  25 points

  on: Is there a.. more ex­act.. way of scor­ing a pre­dic­tor’s cal­ibra­tion?

  It’s im­por­tant to note that ac­cu­racy and cal­ibra­tion are two differ­ent things. I’m men­tion­ing this be­cause the OP asks for cal­ibra­tion met­rics, but sev­eral an­swers so far give ac­cu­racy met­rics. Any proper scor­ing rule is a mea­sure of ac­cu­racy as op­posed to cal­ibra­tion.

  It is pos­si­ble to be very well-cal­ibrated but very in­ac­cu­rate; for ex­am­ple, you might know that it is go­ing to be Mon­day 1/​7th of the time, so you give a prob­a­bil­ity of 1/​7th. Every­one else just knows what day it is. On a cal­ibra­tion graph, you would be perfectly lined up; when you say 1/​7th, the thing hap­pens 1/​7th of the time.

  It is also pos­si­ble to have high ac­cu­racy and poor cal­ibra­tion. Per­haps you can guess coin flips when no one else can, but you are wary of your pre­cog­ni­tive pow­ers, which makes you un­der­con­fi­dent. So, you always place 60% prob­a­bil­ity on the event that ac­tu­ally hap­pens (heads or tails). Your cal­ibra­tion graph is far out of line, but your ac­cu­racy is higher than any­one else.

  In terms of im­prov­ing ra­tio­nal­ity, the in­ter­est­ing thing about cal­ibra­tion is that (as in the pre­cog ex­am­ple) if you know you’re poorly cal­ibrated, you can boost your ac­cu­racy sim­ply by im­prov­ing your cal­ibra­tion. In some sense it is a free im­prove­ment: you don’t need to know any­thing more about the do­main; you get more ac­cu­rate just by know­ing more about your­self (by see­ing a cal­ibra­tion chart and ad­just­ing).

  How­ever, if you just try to be more cal­ibrated with­out any con­cern for ac­cu­racy, you could be like the per­son who says 1/​7th. So, just aiming to do well on a score of cal­ibra­tion is not a good idea. This could be part of the rea­son why cal­ibra­tion charts are pre­sented in­stead of cal­ibra­tion scores. (Another rea­son be­ing that cal­ibra­tion charts help you know how to ad­just to in­crease cal­ibra­tion.)

  That be­ing said, a de­com­po­si­tion of a proper scor­ing rule into com­po­nents in­clud­ing a mea­sure of cal­ibra­tion, like Dark Denego gives, seems like the way to go.

• abramdemski 16 Jan 2019 4:18 UTC

  12 points

  AF

  in reply to: Vaniver's comment on: CDT=EDT=UDT

  I guess, philo­soph­i­cally, I worry that giv­ing the nodes spe­cial types like that pushes peo­ple to­ward think­ing about agents as not-em­bed­ded-in-the-world, think­ing things like “we need to ex­tend Bayes nets to rep­re­sent ac­tions and util­ities, be­cause those are not nor­mal vari­able nodes”. Not that mem­o­ryless carte­sian en­vi­ron­ments are any bet­ter in that re­spect.

• abramdemski 16 Jan 2019 2:45 UTC

  14 points

  in reply to: Jameson Quinn's comment on: CDT=EDT=UDT

  Hrm. I re­al­ize that the post would be com­pre­hen­si­ble to a much wider au­di­ence with a glos­sary, but there’s one level of effort needed for me to write posts like this one, and an­other level needed for posts where I try to be com­pre­hen­si­ble to some­one who lacks all the jar­gon of MIRI-style de­ci­sion the­ory. Ba­si­cally, if I write with a broad au­di­ence in mind, then I’m mod­el­ing all the in­fer­en­tial gaps and ex­plain­ing a lot more de­tails. I would never get to points like the one I’m try­ing to make in this post. (I’ve tried.) Posts like this are pri­mar­ily for the few peo­ple who have kept up with the CDT=EDT se­quence so far, to get my up­dated think­ing in writ­ing in case any­one wants to go through the effort of try­ing to figure out what in the world I mean. To peo­ple who need a glos­sary, I recom­mend search­ing less­wrong and the stan­ford en­cy­clo­pe­dia of philos­o­phy.

[Question] What are the com­po­nents of in­tel­lec­tual hon­esty?

• abramdemski 15 Jan 2019 19:36 UTC

  10 points

  in reply to: MakoYass's comment on: Com­bat vs Nur­ture & Meta-Contrarianism

  I’ve avoided peo­ple/​con­ver­sa­tions on those grounds, but I’m not sure it is the best way to deal with it. And I re­ally do think good in­tel­lec­tual progress can be made at level 2. As Ruby said in the post I’m re­ply­ing to, in­tel­lec­tual de­bate is com­mon in an­a­lytic philos­o­phy, and it does well there.

  Maybe my de­scrip­tion of in­tel­lec­tual de­bate makes you think of all the bad ar­gu­ments-are-sol­diers stuff. Which it should. But, I think there’s some­thing to be said about highly de­vel­oped cul­tures of in­tel­lec­tual de­bate. There are a lot of con­ven­tions which make it work bet­ter, such as a strong norm of be­ing char­i­ta­ble to the other side (which, in in­tel­lec­tual-de­bate cul­ture, means an ex­pec­ta­tion that peo­ple will call you out for be­ing un­char­i­ta­ble). This sort of simu­lates level 3 within level 2.

  As for level 1, you might be able to de­velop some em­pa­thy for it at times when you feel par­tic­u­larly vuln­er­a­ble and need peo­ple to do some­thing to af­firm your be­long­ing­ness in a group or con­ver­sa­tion. Keep an eye out for times when you ap­pre­ci­ate level-one be­hav­ior from oth­ers, times when you would have ap­pre­ci­ated some level-one com­fort, or times when other peo­ple en­gage in level one (and de­cide whether it was helpful in the situ­a­tion). It’s nice when we can get to a place where no one’s ego is on the line when they offer ideas, but some­times it just is. Ig­nor­ing it doesn’t make it go away, it just makes you man­age it in­eptly. My guess is that you are in­volved with more level one situ­a­tions than you think, and would en­dorse some of it.

• abramdemski 14 Jan 2019 7:52 UTC

  2 points

  AF

  in reply to: Diffractor's comment on: CDT Dutch Book

  (lightly ed­ited ver­sion of my origi­nal email re­ply to above com­ment; note that Diffrac­tor was origi­nally re­ply­ing to a ver­sion of the Dutch-book which didn’t yet call out the fact that it re­quired an as­sump­tion of nonzero prob­a­bil­ity on ac­tions.)

  I agree that this Dutch-book ar­gu­ment won’t touch prob­a­bil­ity zero ac­tions, but my think­ing is that it re­ally should ap­ply in gen­eral to ac­tions whose prob­a­bil­ity is bounded away from zero (in some fairly broad set­ting). I’m happy to re­quire an ep­silon-ex­plo­ra­tion as­sump­tion to get the con­clu­sion.

  Your thought ex­per­i­ment raises the is­sue of how to en­sure in gen­eral that adding bets to a de­ci­sion prob­lem doesn’t change the de­ci­sions made. One thought I had was to make the bets always smaller than the differ­ence in util­ities. Per­haps smaller Dutch-books are in some sense less con­cern­ing, but as long as they don’t van­ish to in­finites­i­mal, seems le­git. A bet that’s de­sir­able at one scale is de­sir­able at an­other. But scal­ing down bets may not suffice in gen­eral. Per­haps a bet-bal­anc­ing scheme to en­sure that noth­ing changes the com­par­a­tive de­sir­a­bil­ity of ac­tions as the de­ci­sion is made?

  For your cos­mic ray prob­lem, what about:

  You didn’t spec­ify the prob­a­bil­ity of a cos­mic ray. I sup­pose it should have prob­a­bil­ity higher than the prob­a­bil­ity of ex­plo­ra­tion. Let’s say 1/​mil­lion for cos­mic ray, 1/​billion for ex­plo­ra­tion.

  Be­fore the agent makes the de­ci­sion, it can be given the op­tion to lose .01 util if it goes right, in ex­change for +.02 utils if it goes right & cos­mic ray. This will be ac­cepted (by ei­ther a CDT agent or EDT agent), be­cause it is worth ap­prox­i­mately +.01 util con­di­tioned on go­ing right, since cos­mic ray is al­most cer­tain in that case.

  Then, while mak­ing the de­ci­sion, cos­mic ray con­di­tioned on go­ing right looks very un­likely in terms of CDT’s causal ex­pec­ta­tions. We give the agent the op­tion of get­ting .001 util if it goes right, if it also agrees to lose .02 con­di­tioned on go­ing right & cos­mic ray.

  CDT agrees to both bets, and so loses money upon go­ing right.

  Ah, that’s not a very good money pump. I want it to lose money no mat­ter what. Let’s try again:

  Be­fore de­ci­sion: op­tion to lose 1 mil­lionth of a util in ex­change for 2 utils if right&ray.

  Dur­ing de­ci­sion: op­tion to gain .1 mil­lionth util in ex­change for −2 util if right&ray.

  That should do it. CDT loses .9 mil­lionth of a util, with noth­ing gained. And the trick is al­most the same as my dutch book for death in dam­as­cus. I think this should gen­er­al­ize well.

  The amounts of money lost in the Dutch Book get very small, but that’s fine.

CDT=EDT=UDT

[Question] When is CDT Dutch-Book­able?

• abramdemski 13 Jan 2019 8:28 UTC

  4 points

  AF

  in reply to: ESRogs's comment on: CDT Dutch Book

  “The ex­pec­ta­tions should be equal for ac­tions with nonzero prob­a­bil­ity”—this means a CDT agent should have equal causal ex­pec­ta­tions for any ac­tion taken with nonzero prob­a­bil­ity, and EDT agents should similarly have equal ev­i­den­tial ex­pec­ta­tions. Ac­tu­ally, I should re­vise my state­ment to be more care­ful: in the case of ep­silon-ex­plor­ing agents, the con­di­tion is >ep­silon rather than >0. In any case, my state­ment there isn’t about ev­i­den­tial and causal ex­pec­ta­tions be­ing equal to each other, but rather about one of them be­ing con­ver­sant across (suffi­ciently prob­a­ble) ac­tions.

  “differ­ing coun­ter­fac­tual and ev­i­den­tial ex­pec­ta­tions are smoothly more and more ten­able as ac­tions be­come less and less prob­a­ble”—this means that the amount we can take from a CDT agent through a Dutch Book, for an ac­tion which is given a differ­ent ca­sual ex­pec­ta­tion than ev­i­den­tial ex­pec­ta­tion, smoothly re­duces as the prob­a­bil­ity of an ac­tion goes to zero. In that state­ment, I was as­sum­ing you hold the differ­ence be­tween ev­i­den­tial and causal ex­pec­ta­tions con­stant add you re­duce the prob­a­bil­ity of the ac­tion. Other­wise it’s not nec­es­sar­ily true.

CDT Dutch Book

• abramdemski 12 Jan 2019 23:11 UTC

  9 points

  in reply to: Wei_Dai's comment on: Com­bat vs Nur­ture & Meta-Contrarianism

  I think it’s usu­ally a good idea over­all, but there is a less co­op­er­a­tive con­ver­sa­tional tac­tic which tries to mas­quer­ade as this: list­ing a num­ber of plau­si­ble straw-men in or­der to cre­ate the ap­pear­ance that all pos­si­ble in­ter­pre­ta­tions of what the other per­son is say­ing are bad. (Feels like from the in­side: all pos­si­ble in­ter­pre­ta­tions are bad; i’ll demon­strate it ex­haus­tively...)

  It’s not com­pletely ter­rible, be­cause even this com­bat­ive ver­sion of the con­ver­sa­tional move opens up the op­por­tu­nity for the other per­son to point out the (n+1)th in­ter­pre­ta­tion which hasn’t been enu­mer­ated.

  You can try to differ­en­ti­ate your­self from this via tone (by not sound­ing like you’re try­ing to ar­gue against the other per­son in ask­ing the ques­tion), but, this will only be some­what suc­cess­ful since some­one try­ing to make the less co­op­er­a­tive move will also try to sound like they’re hon­estly try­ing to un­der­stand.