johnswentworth

• johnswentworth 13 Oct 2019 16:21 UTC

  9 points

  on: Pre­dic­tion Mar­kets Don’t Re­veal The Territory

  This is ba­si­cally the same prob­lem as Gears vs Be­hav­ior, spe­cial­ized to the con­text of pre­dic­tion mar­kets. To a large ex­tent, we can use pre­dic­tion mar­kets to pull out in­sights into sys­tem gears us­ing tricks similar to those dis­cussed in that piece. In par­tic­u­lar, causal mod­els are eas­ily adapted to pre­dic­tion mar­kets: just use con­di­tional bets, which only ac­ti­vate when cer­tain con­di­tions are satis­fied. Robin Han­son talks about these fairly of­ten; they’re cen­tral to a lot of his ideas about pre­dic­tion-mar­ket-driven de­ci­sion-mak­ing sys­tems (see e.g. here).

• johnswentworth 11 Oct 2019 3:04 UTC

  2 points

  in reply to: An1lam's comment on: Gears vs Behavior

  Nice links. I ac­tu­ally stopped fol­low­ing deep learn­ing for a few years, and very re­cently started pay­ing at­ten­tion again as the new gen­er­a­tion of prob­a­bil­is­tic pro­gram­ming lan­guages came along (I’m par­tic­u­larly im­pressed with pyro). Those tools are a ma­jor step for­ward for learn­ing causal struc­ture.

  I’d also recom­mend this re­cent pa­per by Fris­ton (the pre­dic­tive pro­cess­ing guy). I might write up a re­view of it soon­ish; it’s a re­ally nice piece of math/​al­gorithm for learn­ing causal struc­ture, again us­ing the same ML tools.

• johnswentworth 9 Oct 2019 21:56 UTC

  LW: 2 AF: 1

  AF

  in reply to: Charlie Steiner's comment on: Char­ac­ter­iz­ing Real-World Agents as a Re­search Meta-Strategy

  I think it will turn out that, with the right no­tion of ab­strac­tion, the un­der­de­ter­mi­na­tion is much less se­vere than it looks at first. In par­tic­u­lar, I don’t think ab­strac­tion is en­tirely de­scribed by a pareto curve of in­for­ma­tion thrown out vs pre­dic­tive power. There are struc­tural crite­ria, and those dra­mat­i­cally cut down the pos­si­bil­ity space.

  Con­sider the Navier-Stokes equa­tions for fluid flow as an ab­strac­tion of (clas­si­cal) molec­u­lar dy­nam­ics. There are other ab­strac­tions which keep around slightly more or slightly less in­for­ma­tion, and make slightly bet­ter or slightly worse pre­dic­tions. But Navier-Stokes is spe­cial among these ab­strac­tions: it has what we might call a “clo­sure” prop­erty. The quan­tities which Navier-Stokes pre­dicts in one fluid cell (av­er­age den­sity & mo­men­tum) can be fully pre­dicted from the cor­re­spond­ing quan­tities in neigh­bor­ing cells plus generic prop­er­ties of the fluid (un­der cer­tain as­sump­tions/​ap­prox­i­ma­tions). By con­trast, imag­ine if we tried to also com­pute the skew or het­eroskedas­tic­ity or other statis­tics of par­ti­cle speeds in each cell. Th­ese would have bizarre in­ter­ac­tions with higher mo­ments, and might not be (ap­prox­i­mately) de­ter­minis­ti­cally pre­dictable at all with­out in­tro­duc­ing even more in­for­ma­tion in each cell. Go­ing the other di­rec­tion, imag­ine we throw out info about den­sity & mo­men­tum in some of the cells. Then that throws off ev­ery­thing else, and sud­denly our whole fluid model needs to track mul­ti­ple pos­si­ble flows.

  So there are “nat­u­ral” lev­els of ab­strac­tion where we keep around ex­actly the quan­tities rele­vant to pre­dic­tion of the other quan­tities. Part of what I’m work­ing on is char­ac­ter­iz­ing these ab­strac­tions: for any given ground-level sys­tem, how can we de­ter­mine which such ab­strac­tions ex­ist? Also, is this the right for­mu­la­tion of a “nat­u­ral” ab­strac­tion, or is there a more/​less gen­eral crite­ria which bet­ter cap­tures our in­tu­itions?

  All this leads into mod­el­ling hu­mans. I ex­pect that there is such a nat­u­ral level of ab­strac­tion which cor­re­sponds to our usual no­tion of “hu­man”, and speci­fi­cally hu­mans as agents. I also ex­pect that this nat­u­ral ab­strac­tion is an agenty model, with “wants” build into it. I do not think that there are a large num­ber of “nearby” nat­u­ral ab­strac­tions.

• johnswentworth 9 Oct 2019 21:26 UTC

  2 points

  in reply to: Xodarap's comment on: Why Subagents?

  Wouldn’t the Hahn em­bed­ding the­o­rem re­sult in a rank­ing of the sub­agents them­selves, rather than re­quiring unan­i­mous agree­ment? Whichever sub­agent cor­re­sponds to the “largest in­fini­ties” (in the sense of or­di­nals) makes its choice, the choice of the next agent only mat­ters if that first sub­agent is in­differ­ent, and so on down the line.

  Any­way, I find the gen­eral idea here in­ter­est­ing. As­sum­ing a group struc­ture seems un­re­al­is­tic as a start­ing point, but there’s a bunch of the­o­rems of the form “any abelian op­er­a­tion with prop­er­ties X, Y, Z is equiv­a­lent to real/​vec­tor ad­di­tion”, so it might not be an is­sue.

• johnswentworth 9 Oct 2019 5:16 UTC

  12 points

  on: Reflec­tions on Premium Poker Tools: Part 2 - De­cid­ing to call it quits

  One thing you might look into is sel­l­ing what you have—there’s a few mar­ket­places for tech pro­jects that haven’t re­ally taken off (e.g. I think flippa is the biggest?). If your product re­ally is worth a de­cent chunk of money, this would at least give you a mar­ket price on the busi­ness as a whole.

• johnswentworth 9 Oct 2019 2:47 UTC

  LW: 4 AF: 2

  AF

  in reply to: Grue_Slinky's comment on: AI Align­ment Open Thread Oc­to­ber 2019

  Oh no, not you too. It was bad enough with just Bena.

• johnswentworth 8 Oct 2019 19:39 UTC

  LW: 6 AF: 3

  AF

  in reply to: Charlie Steiner's comment on: Char­ac­ter­iz­ing Real-World Agents as a Re­search Meta-Strategy

  I was ac­tu­ally go­ing to leave a com­ment on this topic on your last post (which btw I liked, I wish more peo­ple dis­cussed the is­sues in it), but it didn’t seem quite close enough to the topic of that post. So here it is.

  Speci­fi­cally, the idea that there is no “the” wants and on­tol­ogy of e. coli

  This, I think, is the key. My (as-yet-in­com­plete) main an­swer is in “Embed­ded Naive Bayes”: there is a com­pletely un­am­bigu­ous sense in which some sys­tems im­ple­ment cer­tain prob­a­bil­is­tic world-mod­els and other sys­tems do not. Fur­ther­more, the no­tion is sta­ble un­der ap­prox­i­ma­tion: sys­tems which ap­prox­i­mately satisfy the rele­vant func­tional equa­tions use these ap­prox­i­mate world-mod­els. The up­shot is that it is pos­si­ble (at least some­times) to ob­jec­tively, un­am­bigu­ously say that a sys­tem mod­els the world us­ing a par­tic­u­lar on­tol­ogy.

  But it will re­quire abstraction

  Yup. Thus “Embed­ded Agency via Ab­strac­tion”—this has been my plu­ral­ity re­search fo­cus for the past month or so. Think­ing about ab­stract mod­els of ac­tual phys­i­cal sys­tems, I think it’s pretty clear that there are “nat­u­ral” ab­strac­tions in­de­pen­dent of any ob­server, and I’m well on the way to for­mal­iz­ing this use­fully.

  Of course any sort of ab­strac­tion in­volves throw­ing away some pre­dic­tive power, and that’s fine—in­deed that’s ba­si­cally the point of ab­strac­tion. We throw away in­for­ma­tion and only keep what’s needed to pre­dict some­thing of in­ter­est. Navier-Stokes is one ex­am­ple I think about: we throw away the de­tails of micro­scopic mo­tion, and just keep around av­er­aged statis­tics in each lit­tle chunk of space. Navier-Stokes is a “nat­u­ral” level of ab­strac­tion: it’s min­i­mally self-con­tained, with all the info needed to make pre­dic­tions about the bulk statis­tics in each lit­tle chunk of space, but no ad­di­tional info be­yond that.

  Any­way, I’ll prob­a­bly be writ­ing much more about this in the next month or so.

  So if you’re aiming for even­tu­ally tin­ker­ing with hand-coded agen­tial mod­els of hu­mans, one nec­es­sary in­gre­di­ent is go­ing to be tol­er­ance for ab­strac­tion and sub­op­ti­mal pre­dic­tive power.

  Hand-coded mod­els of hu­mans is definitely not some­thing I aim for, but I do think that ab­strac­tion is a nec­es­sary el­e­ment of use­ful mod­els of hu­mans re­gard­less of whether they’re hand-coded. An agenty model of hu­mans is nec­es­sary in or­der to talk about hu­mans want­ing things, which is the whole point of al­ign­ment—and “hu­mans” “want­ing” things only makes sense at a cer­tain level of ab­strac­tion.

• johnswentworth 8 Oct 2019 18:26 UTC

  LW: 8 AF: 4

  AF

  in reply to: Grue_Slinky's comment on: AI Align­ment Open Thread Oc­to­ber 2019

  I think this would be an ex­tremely use­ful ex­er­cise for mul­ti­ple in­de­pen­dent rea­sons:

  • it’s di­rectly at­tempt­ing to teach skills which I do not cur­rently know any re­pro­ducible way to teach/​learn

  • it in­volves look­ing at how break­throughs hap­pened his­tor­i­cally, which is an in­de­pen­dently use­ful meta-strategy

  • it di­rectly in­volves in­ves­ti­gat­ing the in­tu­itions be­hind foun­da­tional ideas rele­vant to the the­ory of agency, and could eas­ily ex­pose al­ter­na­tive views/​in­ter­pre­ta­tions which are more use­ful (in some con­texts) than the usual presentations

Char­ac­ter­iz­ing Real-World Agents as a Re­search Meta-Strategy

• johnswentworth 8 Oct 2019 14:59 UTC

  3 points

  in reply to: Eigil Rischel's comment on: Com­pu­ta­tional Model: Causal Di­a­grams with Symmetry

  The “n==0?” node is in­tended to be a ternary op­er­a­tor; its out­put is n*f(n-1) in the case where n is not 0 (and when n is 0, its out­put is hard­coded to 1).

• johnswentworth 4 Oct 2019 19:57 UTC

  43 points

  on: What are your strate­gies for avoid­ing micro-mis­takes?

  A quote from Wheeler:

  Never make a calcu­la­tion un­til you know the an­swer. Make an es­ti­mate be­fore ev­ery calcu­la­tion, try a sim­ple phys­i­cal ar­gu­ment (sym­me­try! in­var­i­ance! con­ser­va­tion!) be­fore ev­ery deriva­tion, guess the an­swer to ev­ery para­dox and puz­zle.

  When you get into more difficulty math prob­lems, out­side the con­text of a class­room, it’s very easy to push sym­bols around ad-nau­seum with­out mak­ing any for­ward progress. The counter to this is to figure out the in­tu­itive an­swer be­fore start­ing to push sym­bols around.

  When you fol­low this strat­egy, the pro­cess of writ­ing a proof or solv­ing a prob­lem mostly con­sists of re­peat­edly ask­ing “what does my in­tu­ition say here, and how do I trans­late that into the lan­guage of math?” This also gives built-in er­ror checks along the way—if you look at the math, and it doesn’t match what your in­tu­ition says, then some­thing has gone wrong. Either there’s a mis­take in the math, a mis­take in your in­tu­ition, or (most com­mon) a piece was missed in the trans­la­tion.

• johnswentworth 4 Oct 2019 15:49 UTC

  LW: 4 AF: 1

  AF

  in reply to: AlexMennen's comment on: What are we as­sum­ing about util­ity func­tions?

  Let me re­peat back your ar­gu­ment as I un­der­stand it.

  If we have a Bayesian util­ity max­i­miz­ing agent, that’s just a prob­a­bil­is­tic in­fer­ence layer with a VNM util­ity max­i­mizer sit­ting on top of it. So our would-be ar­bi­trageur comes along with a source of “ob­jec­tive” ran­dom­ness, like a quan­tum ran­dom num­ber gen­er­a­tor. The ar­bi­trageur wants to in­ter­act with the VNM layer, so it needs to de­sign bets to which the in­fer­ence layer as­signs some spe­cific prob­a­bil­ity. It does that by us­ing the “ob­jec­tive” ran­dom­ness source in the bet de­sign: just in­cor­po­rate that ran­dom­ness in such a way that the in­fer­ence layer as­signs the prob­a­bil­ities the ar­bi­trageur wants.

  This seems cor­rect in­so­far as it ap­plies. It is a use­ful per­spec­tive, and not one I had thought much about be­fore this, so thanks for bring­ing it in.

  The main is­sue I still don’t see re­solved by this ar­gu­ment is the ar­chi­tec­ture ques­tion. The co­her­ence the­o­rems only say that an agent must act as if they perform Bayesian in­fer­ence and then choose the op­tion with high­est ex­pected value based on those prob­a­bil­ities. In the agent’s ac­tual in­ter­nal ar­chi­tec­ture, there need not be sep­a­rate mod­ules for in­fer­ence and de­ci­sion-mak­ing (a Kal­man filter is one ex­am­ple). If we can’t neatly sep­a­rate the two pieces some­how, then we don’t have a good way to con­struct lot­ter­ies with speci­fied prob­a­bil­ities, so we don’t have a way to treat the agent as a VNM-type agent.

  This di­rectly fol­lows from the origi­nal main is­sue: VNM util­ity the­ory is built on the idea that prob­a­bil­ities live in the en­vi­ron­ment, not in the agent. If there’s a neat sep­a­ra­tion be­tween the agent’s in­fer­ence and de­ci­sion mod­ules, then we can re­define the in­fer­ence mod­ule to be part of the en­vi­ron­ment, but that neat sep­a­ra­tion need not always ex­ist.

  EDIT: Also, I should point out ex­plic­itly that VNM alone doesn’t tell us why we ever ex­pect prob­a­bil­ities to be rele­vant to any­thing in the first place. If we already have a Bayesian ex­pected util­ity max­i­mizer with sep­a­rate in­fer­ence and de­ci­sion mod­ules, then we can model that as an in­fer­ence layer with VNM on top, but then we don’t have a the­o­rem tel­ling us why in­fer­ence lay­ers should mag­i­cally ap­pear in the world.

  Why do we ex­pect (ap­prox­i­mate) ex­pected util­ity max­i­miz­ers to show up in the real world? That’s the main ques­tion co­her­ence the­o­rems an­swer, and VNM can­not an­swer that ques­tion un­less all of the prob­a­bil­ities in­volved are on­tolog­i­cally fun­da­men­tal.

• johnswentworth 3 Oct 2019 14:59 UTC

  LW: 6 AF: 3

  AF

  in reply to: AlexMennen's comment on: What are we as­sum­ing about util­ity func­tions?

  I would ar­gue that in­de­pen­dence of ir­rele­vant al­ter­na­tives is not a real co­her­ence crite­rion. It looks like one at first glance: if it’s vi­o­lated, then you get an Allais Para­dox-type situ­a­tion where some­one pays to throw a switch and then pays to throw it back. The prob­lem is, the “ar­bi­trage” of throw­ing the switch back and forth hinges on the as­sump­tion that the stated prob­a­bil­ities are ob­jec­tively cor­rect. It’s en­tirely pos­si­ble for some­one to come along who be­lieves that throw­ing the switch changes the prob­a­bil­ities in a way that makes it a good deal. Then there’s no real ar­bi­trage, it just comes down to whose prob­a­bil­ities bet­ter match the out­comes.

  My in­tu­ition for this not be­ing real ar­bi­trage comes from fi­nance. In fi­nance, we’d call it “statis­ti­cal ar­bi­trage”: it only works if the prob­a­bil­ities are cor­rect. The ma­jor les­son of the col­lapse of Long Term Cap­i­tal Man­age­ment in the 90′s is that statis­ti­cal ar­bi­trage is definitely not real ar­bi­trage. The whole point of true ar­bi­trage is that it does not de­pend on your statis­ti­cal model be­ing cor­rect .

  This di­rectly leads to the differ­ence be­tween VNM and Bayesian ex­pected util­ity max­i­miza­tion. In VNM, agents have prefer­ences over lot­ter­ies: the prob­a­bil­ities of each out­come are in­puts to the prefer­ence func­tion. In Bayesian ex­pected util­ity max­i­miza­tion, the only in­puts to the prefer­ence func­tion are the choices available to the agent—figur­ing out the prob­a­bil­ities of each out­come un­der each choice is the agent’s job.

  (I do agree that we can set up situ­a­tions where ob­jec­tively cor­rect prob­a­bil­ities are a rea­son­able model, e.g. in a cas­ino, but the point of co­her­ence the­o­rems is to be pretty gen­er­ally ap­pli­ca­ble. A the­o­rem only rele­vant to cas­inos isn’t all that in­ter­est­ing.)

• johnswentworth 2 Oct 2019 21:43 UTC

  LW: 2 AF: 1

  AF

  in reply to: romeostevensit's comment on: What are we as­sum­ing about util­ity func­tions?

  Can you give an ex­am­ple?

• johnswentworth 2 Oct 2019 18:09 UTC

  LW: 18 AF: 5

  AF

  on: What are we as­sum­ing about util­ity func­tions?

  In par­tic­u­lar, the co­her­ence ar­gu­ments and other pres­sures that move agents to­ward VNM seem to roughly scale with ca­pa­bil­ities.

  One nit I keep pick­ing when­ever it comes up: VNM is not re­ally a co­her­ence the­o­rem. The VNM util­ity the­o­rem op­er­ates from four ax­ioms, and only two of those four are rele­vant to co­her­ence. The main prob­lem is that the ax­ioms rele­vant to co­her­ence (acyclic­ity and com­plete­ness) do not say any­thing at all about prob­a­bil­ity and the role that it plays—the “ex­pected” part of “ex­pected util­ity” does not arise from a co­her­ence/​ex­ploita­bil­ity/​pareto op­ti­mal­ity con­di­tion in the VNM for­mu­la­tion of util­ity.

  The ac­tual co­her­ence the­o­rems which un­der­pin Bayesian ex­pected util­ity max­i­miza­tion are things like Dutch book the­o­rems, Wald’s com­plete class the­o­rem, the fun­da­men­tal the­o­rem of as­set pric­ing, and prob­a­bly oth­ers.

  Why does this nit­pick mat­ter? Three rea­sons:

  • In my ex­pe­rience, most peo­ple who ob­ject to the use of util­ities have only en­coun­tered VNM, and cor­rectly point out prob­lems with VNM which do not ap­ply to the real co­her­ence the­o­rems.

  • VNM util­ity stipu­lates that agents have prefer­ences over “lot­ter­ies” with known, ob­jec­tive prob­a­bil­ities of each out­come. The prob­a­bil­ities are as­sumed to be ob­jec­tively known from the start. The Bayesian co­her­ence the­o­rems do not as­sume prob­a­bil­ities from the start; they de­rive prob­a­bil­ities from the co­her­ence crite­ria, and those prob­a­bil­ities are spe­cific to the agent.

  • Be­cause VNM is not re­ally a co­her­ence the­o­rem, I do not ex­pect agent-like sys­tems in the wild to be pushed to­ward VNM ex­pected util­ity max­i­miza­tion. I ex­pect them to be pushed to­ward Bayesian ex­pected util­ity max­i­miza­tion.

• johnswentworth 1 Oct 2019 17:15 UTC

  2 points

  in reply to: crabman's comment on: What fund­ing sources ex­ist for tech­ni­cal AI safety re­search?

  That is not the main fo­cus of the ques­tion, but you’re wel­come to leave an an­swer with sug­ges­tions in that space. It is “fund­ing”, in some sense.

[Question] What fund­ing sources ex­ist for tech­ni­cal AI safety re­search?

• johnswentworth 26 Sep 2019 20:55 UTC

  7 points

  in reply to: jkaufman's comment on: Honor­ing Petrov Day on LessWrong, in 2019

  I dunno, one life seems like a pretty ex­pen­sive trade for the home­page stay­ing up for a day. I bet a po­ten­tial buyer could shop around and ob­tain launch codes for half a life.

  Not say­ing I’d per­son­ally give up my launch code at the very rea­son­able cost of $836. But some­one could prob­a­bly be found. Espe­cially if the buyer some­how found a way to frame some­one else for the launch.

  (Of course, now this com­ment is sit­ting around in plain view of ev­ery­one, the launch codes would have to come from some­one other than me, even ac­count­ing for the fram­ing.)

• johnswentworth 26 Sep 2019 16:41 UTC

  2 points

  on: Free Money at Pre­dic­tIt?

  I’d been check­ing the num­bers on No-side ar­bi­trage for pre­dic­tit’s demo­cratic nom­i­nee and pres­i­dent mar­kets ev­ery cou­ple weeks, but I didn’t re­al­ize that pre­dic­tit frees up your cap­i­tal. How do the de­tails on that work? Is it doc­u­mented some­where?

• johnswentworth 26 Sep 2019 16:09 UTC

  3 points

  in reply to: TekhneMakre's comment on: Ra­tion­al­ity and Levels of Intervention

  The dy­nam­ics in a small group are qual­i­ta­tively differ­ent from whole com­mu­ni­ties. To a large ex­tent, that’s ex­actly why com­mu­nity con­trol is hard/​in­ter­est­ing. Again, Per­sonal to Pri­son Gangs is a good ex­am­ple.