G Gordon Worley III

• G Gordon Worley III 16 Aug 2019 23:43 UTC

  3 points

  in reply to: Matthew Barnett's comment on: Matthew Bar­nett’s Shortform

  I’m some­what sym­pa­thetic to this. You prob­a­bly don’t need the abil­ity, prior to work­ing on AI safety, to already be fa­mil­iar with a wide va­ri­ety of math­e­mat­ics used in ML, by MIRI, etc.. To be spe­cific, I wouldn’t be much con­cerned if you didn’t know cat­e­gory the­ory, more than ba­sic lin­ear alge­bra, how to solve differ­en­tial equa­tions, how to in­te­grate to­gether prob­a­bil­ity dis­tri­bu­tions, or even mul­ti­vari­ate calcu­lus prior to start­ing on AI safety work, but I would be con­cerned if you didn’t have deep ex­pe­rience with writ­ing math­e­mat­i­cal proofs be­yond high school ge­om­e­try (al­though I hear these days they teach ge­om­e­try differ­ently than I learned it—by re-de­riv­ing ev­ery­thing in Ele­ments), say the kind of ex­pe­rience you would get from study­ing grad­u­ate level alge­bra, topol­ogy, mea­sure the­ory, com­bi­na­torics, etc..

  This might also be a bit of mo­ti­vated rea­son­ing on my part, to re­flect Dagon’s com­ments, since I’ve not gone back to study cat­e­gory the­ory since I didn’t learn it in school and I haven’t had spe­cific need for it, but my ex­pe­rience has been that hav­ing solid foun­da­tions in math­e­mat­i­cal rea­son­ing and proof writ­ing is what’s most valuable. The rest can, as you say, be learned lazily, since your needs will be­come ap­par­ent and you’ll have enough math­e­mat­i­cal fluency to find and pur­sue those fields of math­e­mat­ics you may dis­cover you need to know.

• G Gordon Worley III 16 Aug 2019 23:25 UTC

  5 points

  on: Beliefs Are For True Things

  Hold­ing that be­liefs are for true things means that you do not be­lieve things be­cause they are use­ful, be­lieve things be­cause they sound nice, or be­lieve things be­cause you pre­fer them to be true. You be­lieve things that are true (or at least that you be­lieve to be true, which is of­ten the best we can get!).

  This is maybe a sub­tle ob­jec­tion, but I dis­agree with the im­plicit re­jec­tion of util­ity in fa­vor of truth be­ing set up here. Truth is very at­trac­tive to us, and I think this runs deep for rea­sons that don’t much mat­ter here but on which I’ll just say I think it’s be­cause we’re fun­da­men­tally pre­dic­tion er­ror min­i­miz­ers (with some home­o­static feed­back loops thrown in for sur­vival and re­pro­duc­tion pur­poses). But if I had to jus­tify why truth is im­por­tant, I would say it’s be­cause it’s use­ful. If truth were some­how not causally up­stream of mak­ing ac­cu­rate pre­dic­tions about the world (or maybe that’s just what truth means), I don’t think I would care about it, be­cause mak­ing ac­cu­rate pre­dic­tions about the world is re­ally use­ful to get­ting all the other things I care about done.

  Yes, there is a dan­ger that be­falls some peo­ple when they prize util­ity too far above truth that bi­ases them in sub­tle and gross ways that lead them astray and ac­tu­ally work against them by mak­ing them less serve their pur­poses when they’re not look­ing, but there are similar dan­gers when peo­ple pur­sue truth at the ex­pense of use­ful­ness, mostly in the form of op­por­tu­nity costs. I think we all at some point must learn to prize truth over mo­ti­vated rea­son­ing and prefer­ences, for ex­am­ple, but I also think we must learn to prize the util­ity of truth over truth it­self lest we be en­thralled by the Beast of Scrupu­los­ity.

• G Gordon Worley III 13 Aug 2019 19:57 UTC

  2 points

  in reply to: elityre's comment on: Eli’s short­form feed

  I also think it’s rea­son­able to think that mul­ti­ple things may be do­ing on that re­sult in a the­ory of men­tal en­ergy. For ex­am­ple, hy­pothe­ses 1 and 2 could both be true and re­sult in differ­ent causes of similar be­hav­ior. I bring this up be­cause I think of those as two differ­ent things in my ex­pe­rience: be­ing “full up” and need­ing to al­low time for mem­ory con­soli­da­tion where I can still force my at­ten­tion it just doesn’t take in new in­for­ma­tion vs. be­ing un­able to force the di­rec­tion of at­ten­tion gen­er­ally.

• G Gordon Worley III 12 Aug 2019 21:31 UTC

  4 points

  on: Ad­jec­tives from the Fu­ture: The Dangers of Re­sult-based Descriptions

  Maybe it’s be­cause we live in a world full of these “ad­jec­tives from the fu­ture”, but when I think of, for ex­am­ple, a “weight-loss pro­gram” I don’t think the pro­gram will re­sult in weight loss, but rather a pro­gram whose pur­pose is weight loss, whether or not it achieves it. Similarly with the other ex­am­ples: the ad­jec­tive is not de­scribing what it will do, but what the in­tended pur­pose is.

• G Gordon Worley III 12 Aug 2019 21:24 UTC

  2 points

  in reply to: TAG's comment on: Does hu­man choice have to be tran­si­tive in or­der to be ra­tio­nal/​con­sis­tent?

  Right, it does seem that we have found ways, be­ing bounded and ir­ra­tional agents, to get closer to ra­tio­nal­ity by us­ing our bound­ed­ness to pro­tect our­selves from our ir­ra­tional­ity (and vice versa!).

  This seems to be a case of us­ing bound­ed­ness in the form of not be­ing pre­cise, main­tain­ing un­cer­tainty that is not re­solved un­til the last mo­ment, and also prob­a­bly ex­haus­tion (if you try to lead me through a pump af­ter a few steps I’ll give up be­fore you can take too much ad­van­tage of me) to avoid bad re­sults of max­i­miz­ing on ir­ra­tional prefer­ences.

  The op­po­site would be us­ing ir­ra­tional­ity to deal with bound­ed­ness, such as keep­ing things vague so we can some­times still do the right thing even when we’ve made a mis­take in our rea­son­ing about our prefer­ences.

• G Gordon Worley III 12 Aug 2019 21:12 UTC

  9 points

  on: What ex­plana­tory power does Kah­ne­man’s Sys­tem 2 pos­sess?

  As I re­call Kah­ne­man is some­what care­ful to avoid pre­sent­ing S1/​S2 as part of a dual pro­cess the­ory, and in do­ing so nat­u­rally cuts off some of the chance to turn around and use S2 causally up­stream of the things he de­scribes. I think you are cor­rectly see­ing the Kah­ne­man is very care­ful in how he writes, such that S1/​S2 are not gears in his model so much as post hoc pat­terns that act as nice refer­ents to, in his model, iso­lated be­hav­iors that share cer­tain traits with­out hav­ing to pro­pose a unify­ing causal mechanism.

  Nonethe­less, I think we can iden­tify S2 roughly with the neo­cor­tex and S1 roughly with the rest of the brain, and un­der­stand S1/​S2 be­hav­iors as those pri­mar­ily driven by ac­tivity in those parts of the brain. Kah­ne­man just is care­ful, in my rec­ol­lec­tion, to avoid say­ing things like that be­cause there’s no hard proof for it, just in­fer­ence.

• G Gordon Worley III 9 Aug 2019 17:43 UTC

  2 points

  in reply to: Elo's comment on: Scope Insen­si­tivity Judo

  Mostly I think of it in terms of pre­dic­tions and their er­rors. In this ex­am­ple I ex­pected/​pre­dicted the world would look one way and then it looked an­other, and when it looked an­other that seems to have trig­gered a cas­cade of pre­dic­tion er­rors that re­sulted in a pro­cess to try to con­struct new pre­dic­tions that also dredged up old ev­i­dence from mem­ory to be re­con­sid­ered.

• G Gordon Worley III 7 Aug 2019 17:40 UTC

  7 points

  in reply to: John_Maxwell_IV's comment on: Pro­ject Pro­posal: Con­sid­er­a­tions for trad­ing off ca­pa­bil­ities and safety im­pacts of AI research

  I think ML meth­ods are in­suffi­cient for pro­duc­ing AGI, and get­ting to AGI will re­quire one or more changes in paradigm be­fore we have a set of tools that will look like they can pro­duce AGI. From what I can tell the ML com­mu­nity is not work­ing on this, and in­stead pre­fer in­cre­men­tal en­hance­ments to ex­ist­ing al­gorithms.

  Ba­si­cally what I view as needed to make AGI work might be sum­ma­rized as need­ing to de­sign dy­namic feed­back net­works with mem­ory that sup­port on­line learn­ing. What we mostly see out of ML these days are feed­for­ward net­works with offline learn­ing that are static in ex­e­cu­tion and of­ten man­age to work with­out mem­ory, though some do have this. My im­pres­sion is that ex­ist­ing ML al­gorithms are un­sta­ble un­der these kinds of con­di­tions. I ex­pect some­thing like neu­ral net­works will be part of mak­ing it to AGI, and so some cur­rent ML re­search will mat­ter, but mostly we should think of cur­rent ML re­search as be­ing about near-term, nar­row ap­pli­ca­tions rather than on the road to AGI.

  That’s at least my opinion based on my un­der­stand­ing of how con­scious­ness works, my be­lief that “gen­eral” re­quires con­scious­ness, and my un­der­stand­ing of the cur­rent state of ML and what it does and does not do that could sup­port con­scious­ness.

• G Gordon Worley III 7 Aug 2019 17:33 UTC

  2 points

  in reply to: capybaralet's comment on: Pro­ject Pro­posal: Con­sid­er­a­tions for trad­ing off ca­pa­bil­ities and safety im­pacts of AI research

  Based on my read­ing of the post it seemed to me that you were con­cerned pri­mar­ily with info-haz­ard risks in ML re­search, not AI re­search in gen­eral; maybe it’s the way you framed it that I took it to be con­tin­gent on ML mat­ter­ing.

• G Gordon Worley III 7 Aug 2019 1:54 UTC

  6 points

  on: G Gor­don Wor­ley III’s Shortform

  So long as short­form is salient for me, might as well do an­other one on a novel (in that I’ve not heard/​seen any­one ex­press it be­fore) idea I have about per­cep­tual con­trol the­ory, min­i­miza­tion of pre­dic­tion er­ror/​con­fu­sion, free en­ergy, and Bud­dhism that I was re­cently re­minded of.

  There is a no­tion within Ma­hayana Bud­dhism of the three poi­sons: ig­no­rance, at­tach­ment (or, I think we could bet­ter term this here, at­trac­tion, for rea­sons that will be­come clear), and aver­sion. This is part of one model of where suffer­ing arises from. Others ex­press these no­tions in other ways, but I want to fo­cus on this way of talk­ing about these root kle­shas (defile­ments, af­flic­tions, mind poi­sons) be­cause I think it has a clear tie in with this other thing that ex­cites me, the idea that the pri­mary thing that neu­rons seek to do is min­i­mize pre­dic­tion er­ror.

  Ig­no­rance, even among the three poi­sons, is gen­er­ally con­sid­ered more fun­da­men­tal, in that ig­no­rance ap­pears first and it gives rise to at­trac­tion and aver­sion (in some mod­els there is fun­da­men­tal ig­no­rance that gives rise to the three poi­sons, mark­ing a sep­a­ra­tion be­tween ig­no­rance as men­tal ac­tivity and ig­no­rance as a re­sult of the phys­i­cal em­bod­i­ment of in­for­ma­tion trans­fer). This looks to me a lot like what per­cep­tual con­trol the­ory pre­dicts if the thing be­ing con­trol­led for is min­i­miza­tion of pre­dic­tion er­ror: there is con­fu­sion about the state of the world, in­for­ma­tion comes in, and this sends a sig­nal within the con­trol sys­tem of neu­rons to ei­ther up or down reg­u­late some­thing. Essen­tially what the three poi­sons de­scribe is what you would ex­pect the world to look like if the mind were pow­ered by con­trol sys­tems try­ing to min­i­mize con­fu­sion/​ig­no­rance, nudg­ing the sys­tem to­ward and away from a set point where pre­dic­tion er­ror is min­i­mized via nega­tive feed­back (and a small bonus, this might help ex­plain why the brain doesn’t tend to get into long-last­ing pos­i­tive feed­back loops: it’s not con­structed for it and be­fore long you trig­ger some­thing else to down-reg­u­late be­cause you vi­o­late its pre­dic­tions).

  It also makes a lot of sense that these would be the root poi­sons. I think we can for­give 1st mil­len­nium Bud­dhists for not dis­cov­er­ing PCT or min­i­miza­tion of pre­dic­tion er­ror di­rectly, but we should not be sur­prised that they iden­ti­fied the men­tal ac­tions this the­ory pre­dicts should be foun­da­tional to the mind and also rec­og­nized that they were foun­da­tional ac­tions to all oth­ers. Else­where, Bud­dhism ex­plic­itly calls out ig­no­rance as the fun­da­men­tal force driv­ing dukkha (suffer­ing), though we prob­a­bly shouldn’t as­sign too many points to (non-Mad­hya­maka) Bud­dhism for notic­ing this since other Bud­dhist the­o­ries don’t make this same claims about at­tach­ment and aver­sion and they are used con­cur­rently in ex­pli­ca­tion of the dharma.

• G Gordon Worley III 7 Aug 2019 1:31 UTC

  6 points

  on: Subagents, neu­ral Tur­ing ma­chines, thought se­lec­tion, and blindspots

  The pro­duc­tion rule model is in­ter­est­ing to me in that it fits well with Michael Com­mons’ no­tion of how de­vel­op­men­tal psy­chol­ogy works. Speci­fi­cally, Com­mons has a for­mal ver­sion of his the­ory that looks a lot like what de­vel­op­men­tal psy­chol­ogy is about is how hu­mans learn new ways to perform more “com­plex” pro­duc­tion rules in that they are the same sort of rules op­er­at­ing on more com­plex types.

• G Gordon Worley III 7 Aug 2019 0:05 UTC

  3 points

  on: Pro­ject Pro­posal: Con­sid­er­a­tions for trad­ing off ca­pa­bil­ities and safety im­pacts of AI research

  One com­pli­cat­ing fac­tor is how much you be­lieve ML con­tributes to ex­is­ten­tial threats. For ex­am­ple, I think the cur­rent ML com­mu­nity is very un­likely to ever pro­duce AGI (<10%) and that AGI will be the re­sult of break throughs from re­searchers in other parts of AI, thus it seems not very im­por­tant to me what cur­rent ML re­searchers think of long-term safety con­cerns. Other analy­ses of the situ­a­tion would re­sult in con­clud­ing differ­ently, though, so this seems like an up­stream ques­tion that must be ad­dressed or at least con­tin­gently de­cided upon be­fore eval­u­at­ing how much it would make sense to pur­sue this line of in­quiry.

• G Gordon Worley III 6 Aug 2019 20:10 UTC

  10 points

  on: G Gor­don Wor­ley III’s Shortform

  I have plans to write this up more fully as a longer post ex­plain­ing the broader ideas with vi­su­als, but I thought I would high­light one that is pretty in­ter­est­ing and try out the new short­form fea­ture at the same time! As such, this is not op­ti­mized for read­abil­ity, has no links, and I don’t try to backup my claims. You’ve been warned!

  Sup­pose you fre­quently found your­self iden­ti­fy­ing with and feel­ing like you were a ho­muncu­lus con­trol­ling your body and mind: there’s a real you buried in­side, and it’s in the driver’s seat. Some­times your mind and body do what “you” want, some­times it doesn’t and this is frus­trat­ing. Plenty of folks reify this in slightly differ­ent ways: rider and elephant, mon­key and ma­chine, pris­oner in cave (or au­di­ence mem­ber in the­ater), and, to a cer­tain ex­tent, vari­a­tions on the S1/​S2 model. In fact, I would pro­pose this is a kind of dual pro­cess the­ory of mind that has you iden­ti­fy­ing with one of the pro­cesses.

  A few claims.

  First, this is a kind of con­stant, low-level dis­so­ci­a­tion. It’s not the kind of high-in­ten­sity dis­so­ci­a­tion we of­ten think of when we use that term, but it’s still a sep­a­ra­tion of sense of self from the phys­i­cal em­bod­i­ment of self.

  Se­cond, this is pro­jec­tion, and thus a psy­cholog­i­cal prob­lem in need of re­solv­ing. There’s noth­ing good about think­ing of your­self this way; it’s a con­fu­sion that may be tem­porar­ily helpful but it’s also some­thing you need to learn to move be­yond via first rein­te­grat­ing the sep­a­rated sense of self and mind/​body.

  Third, peo­ple drawn to the ra­tio­nal­ist com­mu­nity are un­usu­ally likely to be the sort of folks who dis­so­ci­ate and iden­tify with the ho­muncu­lus, S2, the rider, far mode, or what­ever you want to call it. It gives them a world view that says “ah, yes, I know what’s right, but for some rea­son by stupid brain doesn’t do what I want, so let’s learn how to make it do what I want” when this is in fact a con­fu­sion be­cause it’s the very brain that’s “stupid” that’s pro­duc­ing the feel­ing that you think you know what you want!

  To spec­u­late a bit, this might help ex­plain some of the ra­tio­nal­ist/​meta-ra­tio­nal­ist di­vide: ra­tio­nal­ists are still dis­so­ci­at­ing, meta-ra­tio­nal­ists have already rein­te­grated, and as a re­sult we care about very differ­ent things and look at the world differ­ently be­cause of it. That’s very spec­u­la­tive, though, and I have noth­ing other than weak ev­i­dence to back it up.

G Gor­don Wor­ley III’s Shortform

• G Gordon Worley III 6 Aug 2019 19:16 UTC

  6 points

  on: Prefer­ences as an (in­stinc­tive) stance

  This is a great point that I think some­times gets lost on folks, which is why it’s good that you bring it up. To the ex­tent I dis­agree with you on your re­search agenda, for ex­am­ple, it’s dis­agree­ment over what model we use to de­scribe re­al­ity that will be use­ful to our pur­poses, rather than dis­agree­ment over re­al­ity it­self.

• G Gordon Worley III 6 Aug 2019 1:22 UTC

  8 points

  in reply to: Dagon's comment on: Rae­mon’s Short­form Feed

  Good ex­am­ple: the US tried to go met­ric and then can­celed its com­mit­ment.

• G Gordon Worley III 2 Aug 2019 20:07 UTC

  2 points

  in reply to: johnswentworth's comment on: Why Subagents?

  The prob­lem is that the prefer­ences are con­di­tional on in­ter­nal state; they can’t be cap­tured only by look­ing at the ex­ter­nal en­vi­ron­ment.

  I think I wasn’t clear enough about what I meant. I mean to ques­tion speci­fi­cally why ex­clud­ing such so-called “in­ter­nal” state is the right choice. Yes, it’s difficult and in­con­ve­nient to work with that which we can­not ex­ter­nally ob­serve, but I think much of the prob­lem is that our mod­els leave this part of the world out be­cause it can’t be eas­ily ob­served with suffi­cient fidelity (yet). The di­vi­sion be­tween in­ter­nal and ex­ter­nal is some­what ar­bi­trary in that it ex­ists at the limit of our ob­ser­va­tion pow­ers, not gen­er­ally as a nat­u­ral limit of the sys­tem in­de­pen­dent of our knowl­edge of it, so I ques­tion whether it makes sense to then al­low that limit to de­ter­mine the model we use, rather than step­ping back and find­ing a way to make the model larger such that it can in­clude the episte­molog­i­cal limits that cre­ate par­tial prefer­ences as a con­se­quence rather than be­ing on­tolog­i­cally ba­sic to the model.

• G Gordon Worley III 2 Aug 2019 17:30 UTC

  2 points

  on: Why Subagents?

  With re­gards to path de­pen­dence and par­tial prefer­ences, a cer­tain amount of this feels like the model sim­ply failing to fully cap­ture the prefer­ence on the first go. That is, prefer­ences are con­di­tional, i.e. they are con­di­tioned on the en­vi­ron­ment in which they are em­bed­ded, and the sense in which there is par­tial­ity and path de­pen­dence is­sues seems to me to arise en­tirely from par­tial speci­fi­ca­tion, not the prefer­ence be­ing par­tial it­self. Thus I have to won­der, why pur­sue mod­els that deal with par­tial prefer­ences and their is­sues rather than try­ing to build bet­ter mod­els of prefer­ences that bet­ter cap­ture the full com­plex­ity of prefer­ences?

  To a cer­tain ex­tent it feels to me like with par­tial prefer­ences we’re try­ing to hang on to some things that were con­ve­nient about older mod­els while deal­ing with the com­plex­ities of re­al­ity they failed to ad­e­quately model, rather than giv­ing up our hope to patch the old mod­els and look for some­thing bet­ter suited to what we are try­ing to model (yes, I’m re­veal­ing my own prefer­ence here for new mod­els based on what we learned from old mod­els in­stead of in­cre­men­tally im­prov­ing old mod­els).

• G Gordon Worley III 1 Aug 2019 18:47 UTC

  3 points

  on: Mis­take Ver­sus Con­flict The­ory of Against Billion­aire Philanthropy

  As a gen­eral point, I con­sider it worth writ­ing things that tackle an ob­ject level is­sue and show how mis­take the­ory rea­son­ing con­cludes some­thing differ­ent than con­flict the­ory rea­son­ing and how that is differ­ent. I say that be­cause I think most peo­ple are at least a lit­tle bit con­flict the­o­rists. Maybe not about ev­ery­thing, but at least some­times for many peo­ple there will be times they think in terms of con­flict, of us vs. them, of in-group against out-group. And hav­ing some­one provide a well-rea­soned, thought­ful, and gen­er­ous-to-op­po­nents es­say nudg­ing folks to­wards mis­take the­ory by show­ing how it re­ally works on the mar­gin turns folks into be­ing more strongly mis­take the­o­rists or us­ing mis­take the­ory more of­ten.

  My strong claim would be that hu­mans start out con­flict the­o­rists—it’s our “nat­u­ral” state—and it’s only through peo­ple show­ing us an­other way is it pos­si­ble that we can come to an­other po­si­tion. Yes, any writ­ing like this piece by Scott can be used as fuel for re­in­forc­ing a con­flict the­ory per­spec­tive in some peo­ple, but these are also the peo­ple who are likely so strongly con­flict the­o­rists that all ev­i­dence re­in­forces their po­si­tion and there’s no marginal differ­ence from pro­duc­ing some­thing like Scott’s piece or some­thing less char­i­ta­ble, while it does a lot to move peo­ple to­wards a mis­take the­ory per­spec­tive, even if just on the ob­ject level is­sue ad­dressed, and re­peated ex­po­sure to such peo­ple can turn them into net mis­take the­o­rists.

  Could some ideal per­son have done more to con­vert more con­flict the­o­rists to mis­take the­ory on at least this is­sue in an es­say than Scott did in his? Maybe. But I’m sure Scott did the best he could, and I think it’s on net bet­ter that he wrote this than not.

• G Gordon Worley III 30 Jul 2019 20:54 UTC

  3 points

  on: Fo­rum par­ti­ci­pa­tion as a re­search strategy

  This has similarly been my ap­proach. As best I can tell writ­ing pa­pers for aca­demic pub­li­ca­tion is nice but, es­pe­cially in the AI safety space, is not re­ally the best way to con­vey and dis­cuss ideas. Much more im­por­tant seems to be be­ing part of the con­ver­sa­tion about tech­ni­cal ideas, learn­ing from it, and adding to it so oth­ers can do the same. I put some small amount of effort into things out­side FP mostly be­cause I be­lieve it’s a good idea for rep­u­ta­tion effects and spread­ing ideas out­side the fo­rum bub­ble, but not be­cause I think it’s the best way to make in­tel­lec­tual progress.

  It’s also nice be­cause the feed­back loops are shorter. I can com­ment on a post or write my own, have a dis­cus­sion, and then within weeks see the rip­ples of that dis­cus­sion in­fluenc­ing other dis­cus­sions. It helps me feel the im­pact I’m hav­ing, and mo­ti­vates me to keep go­ing.

  Prob­a­bly the only thing su­pe­rior in my mind is do­ing prac­ti­cal work, e.g. build­ing sys­tems that test out ideas. Un­for­tu­nately many of the ideas we talk about in safety are cur­rently ahead of the tech so we don’t know how to build things yet (and for safety sake I think it’s fine to not push on that too hard since I ex­pect it will come on its own any­way), so un­til we are closer to AGI fo­rum par­ti­ci­pa­tion is likely one of the high im­pact ac­tivi­ties one can en­gage in (I’m similarly pos­i­tive about do­ing the face-to-face equiv­a­lent of talk­ing at con­fer­ences and hav­ing con­ver­sa­tions with in­ter­ested folks).