abramdemski

• abramdemski 17 Nov 2019 16:09 UTC

  LW: 6 AF: 4

  AF

  in reply to: rohinmshah's comment on: The Credit As­sign­ment Problem

  Ac­tu­ally, that wasn’t what I was try­ing to say. But, now that I think about it, I think you’re right.

  I was think­ing of the dis­count­ing var­i­ant of REINFORCE as hav­ing a fixed, but rather bad, model as­so­ci­at­ing re­wards with ac­tions: re­wards are tied more with ac­tions nearby. So I was think­ing of it as still two-level, just worse than ac­tor-critic.

  But, al­though the credit as­sign­ment will make mis­takes (a pre­dictable pun­ish­ment which the agent can do noth­ing to avoid will nonethe­less make any ac­tions lead­ing up to the pun­ish­ment less likely in the fu­ture), they should av­er­age out in the long run (those ‘wrongfully pun­ished’ ac­tions should also be ‘wrongfully re­warded’). So it isn’t re­ally right to think it strongly de­pends on the as­sump­tion.

  In­stead, it’s bet­ter to think of it as a true dis­count­ing func­tion. IE, it’s not as as­sump­tion about the struc­ture of con­se­quences; it’s an ex­pres­sion of how much the sys­tem cares about dis­tant re­wards when tak­ing an ac­tion. Un­der this in­ter­pre­ta­tion, REINFORCE in­deed “closes the gra­di­ent gap”—solves the credit as­sign­ment prob­lem w/​o re­stric­tive mod­el­ing as­sump­tions.

  Maybe. It might also me ar­gued that REINFORCE de­pends on some prop­er­ties of the en­vi­ron­ment such as er­god­ic­ity. I’m not that fa­mil­iar with the de­tails.

  But any­way, it now seems like a plau­si­ble coun­terex­am­ple.

• abramdemski 13 Nov 2019 23:22 UTC

  LW: 2 AF: 1

  AF

  in reply to: steve2152's comment on: The Credit As­sign­ment Problem

  The on­line learn­ing con­cep­tual prob­lem (as I un­der­stand your de­scrip­tion of it) says, for ex­am­ple, I can never know whether it was a good idea to have read this book, be­cause maybe it will come in handy 40 years later. Well, this seems to be “solved” in hu­mans by ex­po­nen­tial /​ hy­per­bolic dis­count­ing. It’s not ex­actly epi­sodic, but we’ll more-or-less be able to ret­ro­spec­tively eval­u­ate whether a cog­ni­tive pro­cess worked as de­sired long be­fore death.

  I in­ter­pret you as sug­gest­ing some­thing like what Ro­hin is sug­gest­ing, with a hy­per­bolic func­tion giv­ing the weights.

  It seems (to me) the liter­a­ture es­tab­lishes that our be­hav­ior can be ap­prox­i­mately de­scribed by the hy­per­bolic dis­count­ing rule (in cer­tain cir­cum­stances any­way), but, comes nowhere near es­tab­lish­ing that the mechanism by which we learn looks like this, and in fact has some ev­i­dence against. But that’s a big topic. For a quick ar­gu­ment, I ob­serve that hu­mans are highly ca­pa­ble, and I gen­er­ally ex­pect ac­tor/​critic to be more ca­pa­ble than dumbly as­so­ci­at­ing re­wards with ac­tions via the hy­per­bolic func­tion. That doesn’t mean hu­mans use ac­tor/​critic; the point is that there are a lot of more-so­phis­ti­cated se­tups to ex­plore.

  We do in fact have a model class.

  It’s pos­si­ble that our mod­els are en­tirely sub­servient to in­stru­men­tal stuff (ie, we “learn to think” rather than “think­ing to learn”, which would mean we don’t have the big split which I’m point­ing to—ie, that we solve the credit as­sign­ment prob­lem “di­rectly” some­how, rather than need­ing to learn to do so.

  It seems very rich; in terms of “grain of truth”, well I’m in­clined to think that noth­ing worth know­ing is fun­da­men­tally be­yond hu­man com­pre­hen­sion, ex­cept for con­tin­gent rea­sons like mem­ory and lifes­pan limi­ta­tions (i.e. not be­cause they are not in­com­pat­i­ble with the in­ter­nal data struc­tures). Maybe that’s good enough?

• abramdemski 13 Nov 2019 23:05 UTC

  LW: 2 AF: 1

  AF

  in reply to: Charlie Steiner's comment on: The Credit As­sign­ment Problem

  Not… re­ally? “how can I max­i­mize ac­cu­racy?” is a very liberal agen­tifi­ca­tion of a pro­cess that might be more drily thought of as ask­ing “what is ac­cu­rate?” Your stan­dard se­quence pre­dic­tor isn’t search­ing through epistemic pseudo-ac­tions to find which ones best max­i­mize its ex­pected ac­cu­racy, it’s just fol­low­ing a pre-made plan of epistemic ac­tion that hap­pens to in­crease ac­cu­racy.

  Yeah, I ab­solutely agree with this. My de­scrip­tion that you quoted was over-dra­mat­i­ciz­ing the is­sue.

  Really, what you have is an agent sit­ting on top of non-agen­tic in­fras­truc­ture. The non-agen­tic in­fras­truc­ture is “op­ti­miz­ing” in a broad sense be­cause it fol­lows a gra­di­ent to­ward pre­dic­tive ac­cu­racy, but it is ut­terly my­opic (doesn’t plan ahead to clev­erly max­i­mize ac­cu­racy).

  The point I was mak­ing, stated more ac­cu­rately, is that you (seem­ingly) need this my­opic op­ti­miza­tion as a ‘pro­tected’ sub-part of the agent, which the over­all agent can­not freely ma­nipu­late (since if it could, it would just cor­rupt the policy-learn­ing pro­cess by wire­head­ing).

  Though this does lead to the thought: if you want to put things on equal foot­ing, does this mean you want to de­scribe a rea­soner that searches through epistemic steps/​rules like an agent search­ing through ac­tions/​plans?

  This is more or less how hu­mans already con­ceive of difficult ab­stract rea­son­ing.

  Yeah, my ob­ser­va­tion is that it in­tu­itively seems like highly ca­pa­ble agents need to be able to do that; to that end, it seems like one needs to be able to de­scribe a frame­work where agents at least have that op­tion with­out it lead­ing to cor­rup­tion of the over­all learn­ing pro­cess via the in­stru­men­tal part strate­gi­cally bi­as­ing the epistemic part to make the in­stru­men­tal part look good.

  (Pos­si­bly hu­mans just use a messy solu­tion where the strate­gic bi­as­ing oc­curs but the dam­age is less­ened by limit­ing the ex­tent to which the in­stru­men­tal sys­tem can bias the epistemics—eg, you can’t fully choose what to be­lieve.)

• abramdemski 13 Nov 2019 22:55 UTC

  LW: 2 AF: 1

  AF

  in reply to: Vanessa Kosoy's comment on: The Credit As­sign­ment Problem

  How does that work?

• abramdemski 13 Nov 2019 22:52 UTC

  LW: 2 AF: 1

  AF

  in reply to: Wei_Dai's comment on: The Credit As­sign­ment Problem

  My think­ing is some­what similar to Vanessa’s. I think a full ex­pla­na­tion would re­quire a long post in it­self. It’s re­lated to my re­cent think­ing about UDT and com­mit­ment races. But, here’s one way of ar­gu­ing for the ap­proach in the ab­stract.

  You once asked:

  As­sum­ing that we do want to be pre-ra­tio­nal, how do we move from our cur­rent non-pre-ra­tio­nal state to a pre-ra­tio­nal one? This is some­what similar to the ques­tion of how do we move from our cur­rent non-ra­tio­nal (ac­cord­ing to or­di­nary ra­tio­nal­ity) state to a ra­tio­nal one. Ex­pected util­ity the­ory says that we should act as if we are max­i­miz­ing ex­pected util­ity, but it doesn’t say what we should do if we find our­selves lack­ing a prior and a util­ity func­tion (i.e., if our ac­tual prefer­ences can­not be rep­re­sented as max­i­miz­ing ex­pected util­ity).

  The fact that we don’t have good an­swers for these ques­tions per­haps shouldn’t be con­sid­ered fatal to pre-ra­tio­nal­ity and ra­tio­nal­ity, but it’s trou­bling that lit­tle at­ten­tion has been paid to them, rel­a­tive to defin­ing pre-ra­tio­nal­ity and ra­tio­nal­ity. (Why are ra­tio­nal­ity re­searchers more in­ter­ested in know­ing what ra­tio­nal­ity is, and less in­ter­ested in know­ing how to be ra­tio­nal? Also, BTW, why are there so few ra­tio­nal­ity re­searchers? Why aren’t there hordes of peo­ple in­ter­ested in these is­sues?)

  My con­tention is that ra­tio­nal­ity should be about the up­date pro­cess. It should be about how you ad­just your po­si­tion. We can have ab­stract ra­tio­nal­ity no­tions as a sort of guid­ing star, but we also need to know how to steer based on those.

  Some ex­am­ples:

  • Log­i­cal in­duc­tion can be thought of as the re­sult of perform­ing this trans­form on Bayesi­anism; it de­scribes be­lief states which are not co­her­ent, and gives a ra­tio­nal­ity prin­ci­ple about how to ap­proach co­her­ence—rather than just in­sist­ing that one must some­how ap­proach co­her­ence.

  • Evolu­tion­ary game the­ory is more dy­namic than the Nash story. It con­cerns it­self more di­rectly with the ques­tion of how we get to equil­ibrium. Strate­gies which work bet­ter get copied. We can think about the equil­ibria, as we do in the Nash pic­ture; but, the evolu­tion­ary story also lets us think about non-equil­ibrium situ­a­tions. We can think about at­trac­tors (equil­ibria be­ing point-at­trac­tors, vs or­bits and strange at­trac­tors), and at­trac­tor bas­ins; the prob­a­bil­ity of end­ing up in one basin or an­other; and other such things.

    • How­ever, al­though the model seems good for study­ing the be­hav­ior of evolved crea­tures, there does seem to be some­thing miss­ing for ar­tifi­cial agents learn­ing to play games; we don’t nec­es­sar­ily want to think of there as be­ing a pop­u­la­tion which is se­lected on in that way.

  • The com­plete class the­o­rem de­scribes util­ity-the­o­retic ra­tio­nal­ity as the end point of tak­ing Pareto im­prove­ments. But, we could in­stead think about ra­tio­nal­ity as the pro­cess of tak­ing Pareto im­prove­ments. This lets us think about (semi-)ra­tio­nal agents whose be­hav­ior isn’t de­scribed by max­i­miz­ing a fixed ex­pected util­ity func­tion, but who de­velop one over time. (This model in it­self isn’t so in­ter­est­ing, but we can think about gen­er­al­iz­ing it; for ex­am­ple, by con­sid­er­ing the difficulty of the bar­gain­ing pro­cess—sub­agents shouldn’t just ac­cept any Pareto im­prove­ment offered.)

    • Again, this model has draw­backs. I’m definitely not say­ing that by do­ing this you ar­rive at the ul­ti­mate learn­ing-the­o­retic de­ci­sion the­ory I’d want.

• abramdemski 13 Nov 2019 22:28 UTC

  LW: 4 AF: 2

  AF

  in reply to: rohinmshah's comment on: The Credit As­sign­ment Problem

  You could also have a ver­sion of REINFORCE that doesn’t make the epi­sodic as­sump­tion, where ev­ery time you get a re­ward, you take a policy gra­di­ent step for each of the ac­tions taken so far, with a weight that de­cays as ac­tions go fur­ther back in time. You can’t prove any­thing in­ter­est­ing about this, but you also can’t prove any­thing in­ter­est­ing about ac­tor-critic meth­ods that don’t have epi­sode bound­aries, I think.

  Yeah, you can do this. I ex­pect ac­tor-critic to work bet­ter, be­cause your sug­ges­tion is es­sen­tially a fixed model which says that ac­tions are more rele­vant to tem­po­rally closer re­wards (and that this is the only fac­tor to con­sider).

  I’m not sure how to fur­ther con­vey my sense that this is all very in­ter­est­ing. My model is that you’re like “ok sure” but don’t re­ally see why I’m go­ing on about this.

• abramdemski 13 Nov 2019 21:33 UTC

  LW: 4 AF: 2

  AF

  in reply to: orthonormal's comment on: The Credit As­sign­ment Problem

  Yeah, it’s definitely re­lated. The main thing I want to point out is that Shap­ley val­ues similarly re­quire a model in or­der to calcu­late. So you have to dis­t­in­guish be­tween the prob­lem of calcu­lat­ing a de­tailed dis­tri­bu­tion of credit and be­ing able to as­sign credit “at all”—in ar­tifi­cial neu­ral net­works, back­prop is how you as­sign de­tailed credit, but a loss func­tion is how you get a no­tion of credit at all. Hence, the ques­tion “where do gra­di­ents come from?”—a re­ward func­tion is like a pile of money made from a joint ven­ture; but to ap­ply back­prop or Shap­ley value, you also need a model of coun­ter­fac­tual pay­offs un­der a va­ri­ety of cir­cum­stances. This is a prob­lem, if you don’t have a seper­ate “epistemic” learn­ing pro­cess to provide that model—ie, it’s a prob­lem if you are try­ing to cre­ate one big learn­ing al­gorithm that does ev­ery­thing.

  Speci­fi­cally, you don’t au­to­mat­i­cally know how to

  send re­wards to each con­trib­u­tor pro­por­tional to how much they im­proved the ac­tual group de­ci­sion

  be­cause in the cases I’m in­ter­ested in, ie on­line learn­ing, you don’t have the op­tion of

  re­run­ning it with­out them and see­ing how perfor­mance declines

  -- be­cause you need a model in or­der to re­run.

  But, also, I think there are fur­ther dis­tinc­tions to make. I be­lieve that if you tried to ap­ply Shap­ley value to neu­ral net­works, it would go poorly; and pre­sum­ably there should be a “philo­soph­i­cal” rea­son why this is the case (why Shap­ley value is solv­ing a differ­ent prob­lem than back­prop). I don’t know ex­actly what the rele­vant dis­tinc­tion is.

  (Or maybe Shap­ley value works fine for NN learn­ing; but, I’d be sur­prised.)

• abramdemski 13 Nov 2019 21:21 UTC

  11 points

  in reply to: romeostevensit's comment on: The Credit As­sign­ment Problem

  Yeah, this one was es­pe­cially difficult in that way. I spent a long time try­ing to ar­tic­u­late the idea in a way that made any sense, and kept adding fram­ing con­text to the be­gin­ning to make the stuff closer to what I wanted to say make more sense—the idea that the post was about the credit as­sign­ment al­gorithm came very late in the pro­cess. I definitely agree that rant-mode feels very vuln­er­a­ble to at­tack.

• abramdemski 13 Nov 2019 21:17 UTC

  LW: 2 AF: 1

  AF

  in reply to: Linda Linsefors's comment on: “em­bed­ded self-jus­tifi­ca­tion,” or some­thing like that

  What you call floor for Alpha Go, i.e. the move eval­u­a­tions, are not even bound­aries (in the sense nos­talge­braist define it), that would just be the ob­ject level (no meta at all) policy.

  I think in gen­eral the idea of the ob­ject level policy with no meta isn’t well-defined, if the agent at least does a lit­tle meta all the time. In AlphaGo, it works fine to shut off the meta; but you could imag­ine a sys­tem where shut­ting off the meta would put it in such an ab­nor­mal state (like it’s on drugs) that the ob­served be­hav­ior wouldn’t mean very much in terms of its usual op­er­a­tion. Maybe this is the point you are mak­ing about hu­mans not hav­ing a good floor/​ceiling dis­tinc­tion.

  But, I think we can con­ceive of the “floor” more gen­er­ally. If the ceiling is the fixed struc­ture, e.g. the up­date for the weights, the “floor” is the low­est-level con­tent—e.g. the weights them­selves. Whether think­ing at some meta-level or not, these weights de­ter­mine the fast heuris­tics by which a sys­tem rea­sons.

  I still think some of what nos­talge­braist said about bound­aries seems more like the floor than the ceiling.

  The space “be­tween” the floor and the ceiling in­volves con­structed meta lev­els, which are larger com­pu­ta­tions (ie not just a sin­gle ap­pli­ca­tion of a heuris­tic func­tion), but which are not fixed. This way we can think of the floor/​ceiling spec­trum as small-to-large: the floor is what hap­pens in a very small amount of time; the ceiling is the whole en­tire pro­cess of the al­gorithm (learn­ing and in­ter­act­ing with the world); the “in­te­rior” is any­thing in-be­tween.

  Of course, this makes it sort of triv­ial, in that you could ap­ply the con­cept to any­thing at all. But the main in­ter­est­ing thing is how an agent’s sub­jec­tive ex­pe­rience seems to in­ter­act with floors and ceilings. IE, we can’t ac­cess floors very well be­cause they hap­pen “too quickly”, and be­sides, they’re the thing that we do ev­ery­thing with (it’s difficult to imag­ine what it would mean for a con­scious­ness to have sub­jec­tive “ac­cess to” its neu­rons/​tran­sis­tors). But we can ob­serve the con­se­quences very im­me­di­ately, and re­flect on that. And the fast op­er­a­tions can be ad­justed rel­a­tively easy (e.g. up­dat­ing neu­ral weights). In­ter­me­di­ate-sized com­pu­ta­tional phe­nom­ena can be rea­soned about, and ac­cessed in­ter­ac­tively, “from the out­side” by the rest of the sys­tem. But the whole com­pu­ta­tion can be “rea­soned about but not up­dated” in a sense, and be­comes difficult to ob­serve again (not “from the out­side” the way smaller sub-com­pu­ta­tions can be ob­served).

• abramdemski 13 Nov 2019 20:24 UTC

  2 points

  in reply to: abramdemski's comment on: Meetup Notes: Ole Peters on ergodicity

  I now like the “time vs en­sem­ble” de­scrip­tion bet­ter. I was try­ing to un­der­stand ev­ery­thing com­ing from a Bayesian frame, but ac­tu­ally, all of these ideas are more fre­quen­tist.

  In a Bayesian frame, it’s nat­u­ral to think di­rectly in terms of a de­ci­sion rule. I didn’t think time-av­er­ag­ing was a good de­scrip­tion be­cause I didn’t see a way for an agent to di­rectly re­place en­sem­ble av­er­age with time av­er­age, in or­der to make de­ci­sions:

  • Ensem­ble av­er­ag­ing is the nat­u­ral re­sponse to de­ci­sion-mak­ing un­der un­cer­tainty; you’re av­er­ag­ing over differ­ent pos­si­bil­ities. When you try to time-av­er­age to get rid of your un­cer­tainty, you have to ask “time av­er­age what?”—you don’t know what spe­cific situ­a­tion you’re in.

  • In gen­eral, the ques­tion of how to turn your cur­rent situ­a­tion into a re­peated se­quence for the pur­pose of time-av­er­ag­ing anal­y­sis seems un­der-de­ter­mined (even if you are cer­tain about your pre­sent situ­a­tion). Surely Peters doesn’t want us to use ac­tual time in the anal­y­sis; in ac­tual time, you end up dead and lose all your money, so the time-av­er­age anal­y­sis is triv­ial.

  • Even if you set­tle on a way to turn the situ­a­tion into an iter­ated se­quence, the nec­es­sary limit does not nec­es­sar­ily ex­ist. This is also true of the pos­si­bil­ity-av­er­age, of course (the St Peters­burg Para­dox be­ing a clas­sic ex­am­ple); but it seems eas­ier to get failure in the time-avarage case, be­cause you just need non-con­ver­gence; ie, you don’t need any un­bounded stuff to hap­pen.

  How­ever, all of these points are also true of fre­quen­tism:

  • Fre­quen­tist ap­proaches start from the ob­jec­tive/​ex­ter­nal per­spec­tive rather than the agent’s in­ter­nal un­cer­tainty. They don’t want to define prob­a­bil­ity as the sub­jec­tive view­point; they want prob­a­bil­ity to be defined as limit­ing fre­quen­cies if you re­peated an ex­per­i­ment over and over again. The fact that you don’t have di­rect ac­cess to these is a nat­u­ral con­se­quence of you not hav­ing di­rect ac­cess to ob­jec­tive truth.

  • Even given di­rect ac­cess to ob­jec­tive truth, fre­quen­tist prob­a­bil­ities are still un­der-defined be­cause of the refer­ence class prob­lem—what in­finite se­quence of ex­per­i­ments do you con­ceive of your ex­per­i­ment as part of?

  • And, again, once you se­lect a se­quence, there’s no guaran­tee that a limit ex­ists. Fre­quen­tism has to solve this by pos­tu­lat­ing that limits ex­ist for the kinds of refer­ence classes we want to talk about.

  So, I now think what Ole Peters is work­ing on is fre­quen­tist de­ci­sion the­ory. Pre­vi­ously, the fre­quen­tist/​Bayesian de­bate was about statis­tics and sci­ence, but de­ci­sion the­ory was pre­dom­i­nantly Bayesian. Ole Peters is work­ing out the nat­u­ral the­ory of de­ci­sion mak­ing which fre­quen­tists could/​should have been pur­su­ing. (So, in that sense, it’s much more than just a new ar­gu­ment for kelly bet­ting.)

  De­scribing fre­quen­tist-vs-Bayesian as time-av­er­ag­ing vs pos­si­bil­ity-av­er­ag­ing (aka en­sem­ble-av­er­ag­ing) seems perfectly ap­pro­pri­ate.

  So, on my un­der­stand­ing, Ole’s re­sponse to the three difficul­ties could be:

  • We first un­der­stand the op­ti­mal re­sponse to an ob­jec­tively defined sce­nario; then, once we’ve done that, we can con­cern our­selves with the ques­tion of how to ac­tu­ally be­have given our un­cer­tainty about what situ­a­tion we’re in. This is not try­ing to be a uni­ver­sal for­mula for ra­tio­nal de­ci­sion mak­ing in the same way Bayesi­anism at­tempts to be; you might have to do some hard work to figure out enough about your situ­a­tion in or­der to ap­ply the the­ory.

    • And when we de­sign gen­eral-pur­pose tech­niques, much like when we de­sign statis­ti­cal tests, our ques­tion should be whether given an ob­jec­tive sce­nario the de­ci­sion-mak­ing tech­nique does well—the same as fre­quen­tists want­ing es­ti­mates to be un­bi­ased. Bayesi­ans want de­ci­sions and es­ti­mates to be op­ti­mal given our un­cer­tainty in­stead.

  • As for how to turn your situ­a­tion into an iter­ated game, Ole can bor­row the fre­quen­tist re­sponse of not say­ing much about it.

  • As for the ex­is­tence of a limit, Ole ac­tu­ally says quite a bit about how to fid­dle with the math un­til you’re deal­ing with a quan­tity for which a limit ex­ists. See his lec­ture notes. On page 24 (just be­fore sec­tion 1.3) he talks briefly about find­ing an ap­pro­pri­ate func­tion of your wealth such that you can do the anal­y­sis. Then, sec­tion 2.7 says much more about this.

    • The gen­eral idea is that you have to choose an anal­y­sis which is ap­pro­pri­ate to the dy­nam­ics. Ad­di­tive dy­nam­ics call for ad­di­tive anal­y­sis (ex­am­in­ing the time-av­er­age of wealth). Mul­ti­plica­tive dy­nam­ics call for mul­ti­plica­tive anal­y­sis (ex­am­in­ing the time-av­er­age of growth, as in kelly bet­ting and similar set­tings). Other set­tings call for other func­tions. Mul­ti­plica­tive dy­nam­ics are com­mon in fi­nan­cial the­ory be­cause so much fi­nan­cial the­ory is about in­vest­ment, but if we ex­am­ine fi­nan­cial de­ci­sions for those liv­ing on in­come, then it has to be very differ­ent.

• abramdemski 9 Nov 2019 23:11 UTC

  5 points

  on: Meetup Notes: Ole Peters on ergodicity

  I haven’t read the ma­te­rial ex­ten­sively (I’ve skimmed it), but here’s what I think is wrong with the time-av­er­age-vs-en­sem­ble-av­er­age ar­gu­ment and my at­tempt to steel­man it.

  It seems very plau­si­ble to me that you’re right about the ques­tion-beg­ging na­ture of Peter’s ver­sion of the ar­gu­ment; it seems like by max­i­miz­ing ex­pected growth rate, you’re max­i­miz­ing log wealth.

  But I also think he’s try­ing to point at some­thing real.

  In the pre­sen­ta­tion where he uses the 1.5x/​0.6x bet ex­am­ple, Peters shows how “ex­pected util­ity over time” is an in­creas­ing line (this is the “en­sem­ble av­er­age”—av­er­ag­ing across pos­si­bil­ities at each time), whereas the ac­tual pay­out for any player looks like a straight down­ward line (in log-wealth) if we zoom out over enough iter­a­tions. There’s no funny busi­ness here—yes, he’s tak­ing a log, but that’s just the best way of graph­ing the phe­nomenon. It’s still true that you lose al­most surely if you keep play­ing this game longer and longer.

  This is a real phe­nomenon. But, how do we for­mal­ize an al­ter­na­tive op­ti­miza­tion crite­rion from it? How do we make de­ci­sions in a way which “ag­gre­gates over time rather than over en­sem­ble”? It’s nat­u­ral to try to for­mal­ize some­thing in log-wealth space since that’s where we see a straight line, but as you said, that’s ques­tion-beg­ging.

  Well, a (fairly gen­eral) spe­cial case of log-wealth max­i­miza­tion is the Kelly crite­rion. How do peo­ple jus­tify that? Wikipe­dia’s cur­rent “proof” sec­tion in­cludes a heuris­tic ar­gu­ment which runs roughly as fol­lows:

  • Imag­ine you’re plac­ing bets in the same way a large num­ber of times, N.

  • By the law of large num­bers, the fre­quency of wins and losses ap­prox­i­mately equals their prob­a­bil­ities.

  • Op­ti­mize to­tal wealth at time N un­der the as­sump­tion that the fre­quen­cies equal the prob­a­bil­ities. You get the Kelly crite­rion.

  Now, it’s easy to see this deriva­tion and think “Ah, so the Kelly crite­rion op­ti­mizes your wealth af­ter a large num­ber of steps, whereas ex­pected util­ity only looks one step ahead”. But, this is not at all the case. An ex­pected money max­i­mizer (EMM) think­ing long-term will still take risky bets. Ob­serve that (in the in­vest­ment set­ting in which Kelly works) the EMM strat­egy for a sin­gle step doesn’t de­pend on the amount of money you have—you ei­ther put all your money in the best in­vest­ment, or you keep all of your money be­cause there are no good in­vest­ments. There­fore, the pay­out of the EMM in a sin­gle step is some mul­ti­ple C of the amount of money it be­gins that step with. There­fore, an EMM look­ing one step ahead just val­ues its win­nings at the end of the first step C more—but this doesn’t change its be­hav­ior, since mul­ti­ply­ing ev­ery­thing by C doesn’t change what the max-ex­pec­ta­tion strat­egy will be. Similarly, two-step looka­head only mod­ifies things by $C^2$, and so on. So an EMM look­ing far ahead be­haves just like one max­i­miz­ing its hold­ings in the very next step.

  The trick in the anal­y­sis is the way we re­place a big sum over lots of pos­si­ble ways things could go with a sin­gle “typ­i­cal” out­come. This might ini­tially seem like a mere com­pu­ta­tional con­ve­nience—af­ter all, the vast vast ma­jor­ity of pos­si­ble se­quences have ap­prox­i­mately the ex­pected win/​loss fre­quen­cies. Here, though, it makes all the differ­ence, be­cause it elimi­nates from con­sid­er­a­tion the wor­lds which have the high­est weight in the EMM anal­y­sis—the wor­lds where things to re­ally well and the EMM gets ex­po­nen­tially much money.

  OK, so, is the deriva­tion just a mis­take?

  I think many en­glish-lan­guage jus­tifi­ca­tions of the Kelly crite­rion or log-wealth max­i­miza­tion are mis­lead­ing or out­right wrong. I don’t think we can jus­tify it as an anal­y­sis of the best long-term strat­egy, be­cause the anal­y­sis rules out any se­quence other than those with the most prob­a­ble statis­tics, which isn’t a move mo­ti­vated by long-term anal­y­sis. I don’t think we can even jus­tify it as “time av­er­age rather than en­sem­ble av­er­age” be­cause we’re not time-av­er­ag­ing wealth. In­deed, the whole point is sup­pos­edly to deal with the non-er­godic cases; but non-er­godic sys­tems don’t have unique time-av­er­aged be­hav­ior!

  How­ever, I ul­ti­mately find some­thing con­vinc­ing about the anal­y­sis: namely, from an evolu­tion­ary per­spec­tive, we ex­pect to even­tu­ally find that only (ap­prox­i­mate) log-wealth max­i­miz­ers re­main in the mar­ket (with non-neg­ligible funds).

  This con­clu­sion is perfectly com­pat­i­ble with ex­pected util­ity the­ory as em­bod­ied by the VNM ax­ioms et cetera. It’s an ar­gu­ment that mar­ket en­tities will tend to have util­ity=log(money), at least ap­prox­i­mately, at least in com­mon situ­a­tions which we can ex­pect strate­gies to be op­ti­mized for. More gen­er­ally, there might be an ar­gu­ment that evolved or­ganisms will tend to have util­ity=log(re­sources), for many no­tions of re­sources.

  How­ever, maybe Nas­sim Ni­co­las Taleb would re­buke us for this tepid and timid con­clu­sion. In terms of pure util­ity the­ory, ap­ply­ing a log be­fore tak­ing an ex­pec­ta­tion is a dis­tinc­tion with­out a differ­ence—we were al­lowed any util­ity func­tion we wanted from the start, so re­quiring an ar­bi­trary trans­form means noth­ing. For ex­am­ple, we can “solve” the St. Peters­burg para­dox by claiming our util­ity is the log of money—but we can then re-cre­ate the para­dox by putting all the num­bers in the game through an ex­po­nen­tial func­tion! So what’s the point? We should learn from our past mis­takes, and choose a frame­work which won’t be prone to those same er­rors.

  So, can we steel­man the claims that ex­pected util­ity the­ory is wrong? Can we find a de­ci­sion pro­ce­dure which is con­sis­tent with the Peters’ gen­eral idea, but isn’t just log-wealth max­i­miza­tion?

  Well, let’s look again at the kelly-crite­rion anal­y­sis. Can we make that into a gen­eral-pur­pose de­ci­sion pro­ce­dure? Can we get it to pro­duce re­sults in­com­pat­i­ble with VNM? If so, is the pro­ce­dure at all plau­si­ble?

  As I’ve already men­tioned, there isn’t a clear way to ap­ply the law-of-large-num­bers trick in non-er­godic situ­a­tions, be­cause there is not a unique “typ­i­cal” set of fre­quen­cies which emerges. Can we do any­thing to re­pair the situ­a­tion, though?

  I pro­pose that we max­i­mize me­dian ex­pected value. This gives a no­tion of “typ­i­cal” which does not rely on an ap­pli­ca­tion of the law of large num­bers, so it’s fine if the statis­tics of our se­quence don’t con­verge to a sin­gle unique point. If they do, how­ever, the me­dian will eval­u­ate things from that point. So, it’s a work­able gen­er­al­iza­tion of the prin­ci­ple be­hind Kelly bet­ting.

  The me­dian also re­lates to some­thing men­tioned in the OP:

  I’ve felt vaguely con­fused for a long time about why ex­pected value/​util­ity is the right way to eval­u­ate de­ci­sions; it seems like I might be more strongly in­ter­ested in some­thing like “the 99th per­centile out­come for the over­all util­ity gen­er­ated over my life­time”.

  The me­dian is the 50th per­centile, so there you go.

  Max­i­miz­ing the me­dian in­deed vi­o­lates VNM:

  • It’s dis­con­tin­u­ous. Small differ­ences in prob­a­bil­ity can change the me­dian out­come by a lot. Maybe this isn’t so bad—who re­ally cares about con­ti­nu­ity, any­way? Yeah, seem­ingly small differ­ences in prob­a­bil­ity cre­ate “un­jus­tified” large differ­ences in per­ceived qual­ity of a plan, but only in cir­cum­stances where out­comes are sparse enough that the me­dian is not very “in­formed”.

  • It vi­o­lates in­de­pen­dence, in a more ob­vi­ously con­cern­ing way. A me­dian-max­i­mizer doesn’t care about “out­lier” out­comes. It’s in­differ­ent be­tween the fol­low­ing two plans, which seems ut­terly wrong:

    • A plan with 100% prob­a­bil­ity of get­ting you $100

    • A plan with 60% prob­a­bil­ity of get­ting you $100, and 40% prob­a­bil­ity of get­ting you kil­led.

  Both of these con­cerns be­come neg­ligible as we take a long-term view. The longer into the fu­ture we look, the more out­comes there will be, mak­ing the me­dian more ro­bust to shift­ing prob­a­bil­ities. Similarly, a me­dian-max­i­mizer is in­differ­ent be­tween the two op­tions above, but if you con­sider the iter­ated game, it will strongly pre­fer the global strat­egy of always se­lect­ing the first op­tion.

  Still, I would cer­tainly not pre­fer to op­ti­mize me­dian value my­self, or cre­ate AGI which op­ti­mizes me­dian value. What if there’s a one-shot situ­a­tion which is similar to the 40%-death ex­am­ple? I think I similarly don’t want to max­i­mize the 99th per­centile out­come, al­though this is less clearly ter­rible.

  Can we give an evolu­tion­ary ar­gu­ment for me­dian util­ity, as a gen­er­al­iza­tion of the evolu­tion­ary ar­gu­ment for log util­ity? I don’t think so. The evolu­tion­ary ar­gu­ment re­lies on the law of large num­bers, to say that we’ll al­most surely end up in a world where log-max­i­miz­ers pros­per. There’s no similar ar­gu­ment that we al­most surely end up in the “me­dian world”.

  So, all told:

  • I don’t think there’s a good ar­gu­ment against ex­pec­ta­tion-max­i­miza­tion here.

    • But I do think those who think there is should con­sider me­dian-max­i­miza­tion, as it’s an al­ter­na­tive to ex­pec­ta­tion-max­i­miza­tion which is con­sis­tent with much of the dis­cus­sion here.

  • I ba­si­cally buy the ar­gu­ment that util­ity should be log of money.

  • I don’t think it’s right to de­scribe the whole thing as “time-av­er­age vs en­sem­ble-av­er­age”, and sus­pect some of the “deriva­tions” are ques­tion-beg­ging.

  • I do think there’s an evolu­tion­ary ar­gu­ment which can be un­der­stood from some of the deriva­tions, how­ever.

• abramdemski 9 Nov 2019 20:35 UTC

  2 points

  in reply to: Orborde's comment on: Meetup Notes: Ole Peters on ergodicity

  It seems to me like it’s right. So far as I can tell, the “time-av­er­age vs en­sem­ble av­er­age” ar­gu­ment doesn’t re­ally make sense, but it’s still true that log-wealth max­i­miza­tion is a dis­t­in­guished risk-averse util­ity func­tion with es­pe­cially good prop­er­ties.

  • Ideal­ized mar­kets will evolve to con­tain only Kelly bet­tors, as other strate­gies ei­ther go bust too of­ten or have sub-op­ti­mal growth.

    • BUT, keep in mind we don’t live in such an ideal­ized mar­ket. In re­al­ity, it only makes sense to use this ar­gu­ment to con­clude that fi­nan­cially savvy peo­ple/​in­sti­tu­tions will be ap­prox­i­mate log-wealth max­i­miz­ers—IE, the peo­ple/​or­ga­ni­za­tions with a lot of money. Reg­u­lar peo­ple might be nowhere near log-wealth-max­i­miz­ing, be­cause “go­ing bust” of­ten doesn’t liter­ally mean dy­ing; you can be a failed se­rial startup founder, be­cause you can crash on friends’/​par­ents’ couches be­tween ven­tures, work ba­sic jobs when nec­es­sary, etc.

  • More gen­er­ally, evolved or­ganisms are likely to be ap­prox­i­mately log-re­source max­i­miz­ers. I’m less clear on this ar­gu­ment, but the situ­a­tion seems analo­gous. It there­fore may make sense to sup­pose that hu­mans are ap­prox­i­mate log-re­source max­i­miz­ers.

  (I’m not claiming Peters is nec­es­sar­ily adding any­thing to this anal­y­sis.)

• abramdemski 9 Nov 2019 7:47 UTC

  LW: 6 AF: 3

  AF

  in reply to: ofer's comment on: Defin­ing Myopia

  Sorry for tak­ing so long to re­spond to this one.

  I don’t get the last step in your ar­gu­ment:

  In con­trast, if our learn­ing al­gorithm is some evolu­tion­ary com­pu­ta­tion al­gorithm, the mod­els (in the pop­u­la­tion) in which θ8 hap­pens to be larger are ex­pected to out­perform the other mod­els, in iter­a­tion 2. There­fore, we should ex­pect iter­a­tion 2 to in­crease the av­er­age value of θ8 (over the model pop­u­la­tion).

  Why do those mod­els out­perform? I think you must be imag­in­ing a differ­ent setup, but I’m in­ter­pret­ing your setup as:

  • This is a clas­sifi­ca­tion prob­lem, so, we’re get­ting feed­back on cor­rect la­bels X for some Y.

  • It’s on­line, so we’re do­ing this in se­quence, and learn­ing af­ter each.

  • We keep a pop­u­la­tion of mod­els, which we up­date (per­haps only a lit­tle) af­ter ev­ery train­ing ex­am­ple; pop­u­la­tion mem­bers who pre­dicted the la­bel cor­rectly get a chance to re­pro­duce, and a few pop­u­la­tion mem­bers who didn’t are kil­led off.

  • The over­all pre­dic­tion made by the sys­tem is the av­er­age of all the pre­dic­tions (or some other ag­gre­ga­tion).

  • Large ${\theta }_8$ in­fluences at one time-step will cause pre­dic­tions which make the next time-step eas­ier.

  • So, if the pop­u­la­tion has an abun­dance of high ${\theta }_8$ at one time step, the pop­u­la­tion over­all does bet­ter in the next time step, be­cause it’s eas­ier for ev­ery­one to pre­dict.

  • So, the fre­quency of high ${\theta }_8$ will not be in­creased at all. Just like in gra­di­ent de­scent, there’s no point at which the rele­vant pop­u­la­tion mem­bers are speci­fi­cally re­warded.

  In other words, many mem­bers of the pop­u­la­tion can swoop in and reap the benefits caused by high-${\theta }_8$ mem­bers. So high-${\theta }_8$ car­ri­ers do not speci­fi­cally benefit.

• abramdemski 8 Nov 2019 19:35 UTC

  LW: 8 AF: 4

  AF

  in reply to: rohinmshah's comment on: The Credit As­sign­ment Problem

  Yeah, I pretty strongly think there’s a prob­lem—not nec­es­sar­ily an in­sol­u­ble prob­lem, but, one which has not been con­vinc­ingly solved by any al­gorithm which I’ve seen. I think pre­sen­ta­tions of ML of­ten ob­scure the prob­lem (be­cause it’s not that big a deal in prac­tice—you can of­ten define good enough epi­sode bound­aries or what­not).

  Sup­pose we have a good re­ward func­tion (as is typ­i­cally as­sumed in deep RL). We can just copy the trick in that set­ting, right? But the rest of the post makes it sound like you still think there’s a prob­lem, in that even with that re­ward, you don’t know how to as­sign credit to each in­di­vi­d­ual ac­tion. This is a prob­lem that evolu­tion also has; evolu­tion seemed to man­age it just fine.

  • Yeah, I feel like “match­ing re­wards to ac­tions is hard” is a pretty clear ar­tic­u­la­tion of the prob­lem.

  • I agree that it should be sur­pris­ing, in some sense, that get­ting re­wards isn’t enough. That’s why I wrote a post on it! But why do you think it should be enough? How do we “just copy the trick”??

  • I don’t agree that this is analo­gous to the prob­lem evolu­tion has. If evolu­tion just “re­ceived” the over­all pop­u­la­tion each gen­er­a­tion, and had to figure out which genomes were good/​bad based on that, it would be a more analo­gous situ­a­tion. How­ever, that’s not at all the case. Evolu­tion “re­ceives” a fairly rich vec­tor of which genomes were bet­ter/​worse, each gen­er­a­tion. The analo­gous case for RL would be if you could out­put sev­eral ac­tions each step, rather than just one, and re­ceive feed­back about each. But this is ba­si­cally “ac­cess to coun­ter­fac­tu­als”; to get this, you need a model.

  (Similarly, even if you think ac­tor-critic meth­ods don’t count, surely REINFORCE is one-level learn­ing? It works okay; added bells and whis­tles like crit­ics are im­prove­ments to its sam­ple effi­ciency.)

  No, definitely not, un­less I’m miss­ing some­thing big.

  From page 329 of this draft of Sut­ton & Barto:

  Note that REINFORCE uses the com­plete re­turn from time t, which in­cludes all fu­ture re­wards up un­til the end of the epi­sode. In this sense REINFORCE is a Monte Carlo al­gorithm and is well defined only for the epi­sodic case with all up­dates made in ret­ro­spect af­ter the epi­sode is com­pleted (like the Monte Carlo al­gorithms in Chap­ter 5). This is shown ex­plic­itly in the boxed on the next page.

  So, REINFORCE “solves” the as­sign­ment of re­wards to ac­tions via the blunt de­vice of an epi­sodic as­sump­tion; all re­wards in an epi­sode are grouped with all ac­tions dur­ing that epi­sode. If you ex­pand the epi­sode to in­finity (so as to make no as­sump­tion about epi­sode bound­aries), then you just aren’t learn­ing. This means it’s not ap­pli­ca­ble to the case of an in­tel­li­gence wan­der­ing around and in­ter­act­ing dy­nam­i­cally with a world, where there’s no par­tic­u­lar bound on how the past may re­late to pre­sent re­ward.

  The “model” is thus ex­tremely sim­ple and hard­wired, which makes it seem one-level. But you can’t get away with this if you want to in­ter­act and learn on-line with a re­ally com­plex en­vi­ron­ment.

  Also, since the epi­sodic as­sump­tion is a form of my­opia, REINFORCE is com­pat­i­ble with the con­jec­ture that any gra­di­ents we can ac­tu­ally con­struct are go­ing to in­cen­tivize some form of my­opia.

• abramdemski 8 Nov 2019 17:32 UTC

  LW: 5 AF: 3

  AF

  in reply to: steve2152's comment on: The Credit As­sign­ment Problem

  Yep, I 100% agree that this is rele­vant. The PP/​Fris­ton/​free-en­ergy/​ac­tive-in­fer­ence camp is definitely at least try­ing to “cross the gra­di­ent gap” with a unified the­ory as op­posed to a two-sys­tem solu­tion. How­ever, I’m not sure how to think about it yet.

  • I may be com­pletely wrong, but I have a sense that there’s a dis­tinc­tion be­tween learn­ing and in­fer­ence which plays a similar role; IE, plan­ning is just in­fer­ence, but both plan­ning and in­fer­ence work only be­cause the learn­ing part serves as the sec­ond “pro­tected layer”??

  • It may be that the PP is “more or less” the Bayesian solu­tion; IE, it re­quires a grain of truth to get good re­sults, so it doesn’t re­ally help with the things I’m most in­ter­ested in get­ting out of “cross­ing the gap”.

  • Note that PP clearly tries to im­ple­ment things by push­ing ev­ery­thing into epistemics. On the other hand, I’m mostly dis­cussing what hap­pens when you try to smoosh ev­ery­thing into the in­stru­men­tal sys­tem. So many of my re­marks are not di­rectly rele­vant to PP.

  • I get the sense that Fris­ton might be us­ing the “evolu­tion solu­tion” I men­tioned; so, unify­ing things in a way which kind of lets us talk about evolved agents, but not ar­tifi­cial ones. How­ever, this is ob­vi­ously an over­sim­plifi­ca­tion, be­cause he does pre­sent de­signs for ar­tifi­cial agents based on the ideas.

  Over­all, my cur­rent sense is that PP ob­scures the is­sue I’m in­ter­ested in more than solves it, but it’s not clear.

The Credit As­sign­ment Problem

• abramdemski 7 Nov 2019 23:51 UTC

  3 points

  in reply to: bkam's comment on: The Zet­telkas­ten Method

  Not re­ally? Although I use in­ter­con­nec­tions, I fo­cus a fair amount on the tree-struc­ture part. I would say there’s a some­what cu­ri­ous phe­nomenon where I am able to go “deeper” in anal­y­sis than I would pre­vi­ously (in note­books or work­flowy), but the “shal­low” part of the anal­y­sis isn’t ques­tioned as much as it could be (it be­comes the con­text in which things hap­pen). In a note­book, I might end up re-stat­ing “early” parts of my over­all ar­gu­ment more, and there­fore re­fin­ing them more.

  I have definitely had the ex­pe­rience of reach­ing a con­clu­sion fairly strongly in Zet­telkas­ten and then hav­ing trou­ble ar­tic­u­lat­ing it to other peo­ple. My un­der­stand­ing of the situ­a­tion is that I’ve built up a lot of con­text of which ques­tions are worth ask­ing, how to ask them, which ex­am­ples are most in­ter­est­ing, etc. So there’s a longer in­fer­en­tial dis­tance. BUT, it’s also a bad sign for the con­clu­sion. The con­text I’ve built up is more prob­a­bly shaky if I can’t ar­tic­u­late it very well.

• abramdemski 6 Nov 2019 20:00 UTC

  8 points

  in reply to: bkam's comment on: The Zet­telkas­ten Method

  My worry was es­sen­tially me­dia-makes-mes­sage style. Luh­mann’s so­ciolog­i­cal the­o­ries were sprawl­ing in­ter­con­nected webs. (I have not read him at all; this is just my im­pres­sion.) This is not nec­es­sar­ily be­cause the re­al­ity he was look­ing at is best un­der­stood in that form. Also, his the­ory of so­ciol­ogy has some­thing to do with sys­tems in­ter­act­ing with each other through com­mu­ni­ca­tion bot­tle­necks (?? again, I have not re­ally read him), which he ex­plic­itly re­lates to Zet­telkas­ten.

  Re­lat­edly, Paul Chris­ti­ano uses a work­flowy-type out­lin­ing tool ex­ten­sively, and his the­ory of AI safety promi­nently fea­tures hi­er­ar­chi­cal tree struc­tures.

• abramdemski 4 Nov 2019 17:40 UTC

  9 points

  on: Dream­ing of Poli­ti­cal Bayescraft

  Any time you find your­self be­ing tempted to be loyal to an idea, it turns out that what you should ac­tu­ally be loyal to is what­ever un­der­ly­ing fea­ture of hu­man psy­chol­ogy makes the idea look like a good idea; that way, you’ll find it eas­ier to fuck­ing up­date when it turns out that the im­ple­men­ta­tion of your fa­vorite idea isn’t as fun as you ex­pected!

  I agree that there’s an im­por­tant skill here, but I also want to point out that this seems to tip in a par­tic­u­lar di­rec­tion which may be con­cern­ing.

  Ben Hoff­man writes about au­then­tic­ity vs ac­cu­racy.

  • An au­then­tic­ity-ori­ented per­son thinks of hon­esty as be­ing true to what you’re feel­ing right now. Quick an­swers from the gut are more hon­est. Care­ful con­sid­er­a­tion be­fore speak­ing is a sign of dishon­esty. Mak­ing a promise and later break­ing it isn’t dishon­est if you re­ally meant the promise when you made it!

  • An ac­cu­racy-ori­ented per­son thinks of hon­esty as mak­ing a real effort to tell the truth. Quick an­swers are a sign that you’re not do­ing that; long pauses be­fore speak­ing are a sign that you are. It’s not just about say­ing what you re­ally be­lieve; mak­ing a fac­tual er­ror when you could have avoided it if you had been more care­ful is al­most the same as pur­pose­fully ly­ing (es­pe­cially given con­cerns about mo­ti­vated cog­ni­tion).

  Authen­tic­ity and ac­cu­racy are both valuable, and it would be best to rec­on­cile them. But, my con­cern is that your ad­vice against be­ing loyal to an idea tips things away from ac­cu­racy. If you have a knee-jerk re­ac­tion to be loyal to the gen­er­a­tors of an idea rather than the idea it­self, it seems to me like you’re go­ing to make some slips to­ward the mak­ing-a-promise-and-break­ing-it-isn’t-dishon­est-if-you-meant-it di­rec­tion which you wouldn’t re­flec­tively en­dorse if you con­sid­ered it more care­fully.

• abramdemski 3 Nov 2019 7:15 UTC

  LW: 2 AF: 1

  AF

  in reply to: MakoYass's comment on: The Parable of Pre­dict-O-Matic

  I guess ‘self-fulfilling prophecy’ is a bit long and awk­ward. Some­times ‘basilisk’ is thrown around, but, speci­fi­cally for nega­tive cases (self-fulfilling-and-bad). But, are you try­ing to name some­thing slightly differ­ent (per­haps broader or nar­rower) than self-fulfilling prophecy points at?

  I find I don’t like ‘stipu­la­tion’; that has the con­no­ta­tion of com­mand, for me (like, if I tell you to do some­thing).