steve2152

• steve2152 11 Sep 2019 11:30 UTC

  4 points

  AF

  in reply to: rohinmshah's comment on: Two senses of “op­ti­mizer”

  If the su­per-pow­er­ful SAT solver thing finds the plans but doesn’t ex­e­cute them, would you still lump it with op­ti­mizer_2? (I know it’s just ter­minol­ogy and there’s no right an­swer, but I’m just cu­ri­ous about what cat­e­gories you find nat­u­ral.)

  (BTW this is more-or-less a de­scrip­tion of my cur­rent Grand Vi­sion For AGI Safety, where the “dy­nam­ics of the world” are dis­cov­ered by self-su­per­vised learn­ing, and the search pro­cess (and much else) is TBD.)

• steve2152 5 Sep 2019 1:09 UTC

  1 point

  in reply to: John_Maxwell's comment on: Self-su­per­vised learn­ing & ma­nipu­la­tive predictions

  This is great, thanks again for your time and thought­ful com­men­tary!

  RE “I’m not en­tirely con­vinced that pre­dic­tions should be made in a way that’s com­pletely di­vorced from their effects on the world.”: My vi­sion is to make a non-agen­tial ques­tion-an­swer­ing AGI, thus avoid­ing value al­ign­ment. I don’t claim that this is definitely the One Right An­swer To AGI Safety (see “4. Give up, and just make an agent with value-al­igned goals” in the post), but I think it is a plau­si­ble (and ne­glected) can­di­date an­swer. See also my post In defense of or­a­cle (tool) AI re­search for why I think it would solve the AGI safety prob­lem.

  If an AGI ap­plies its in­tel­li­gence and world model to its own out­put, choos­ing that out­put partly for its down­stream effects as pre­dicted by the model, then I say it’s a goal-seek­ing agent. In this case, we need to solve value al­ign­ment—even if the goal is as sim­ple as “an­swer my ques­tion”. (We would need to make sure that the goal is what it’s sup­posed to be, as op­posed to a proxy goal, or a weird alien in­ter­pre­ta­tion where rewiring the op­er­a­tor’s brain counts as “an­swer my ques­tion”.) Again, I’m not op­posed to build­ing agents af­ter solv­ing value al­ign­ment, but we haven’t solved value al­ign­ment yet, and thus it’s worth ex­plor­ing the other op­tion: build a non-agent which does not in­tel­li­gently model the down­stream effects of its out­put at all (or if it does model it in­ci­den­tally, to not do any­thing with that in­for­ma­tion).

  In­ter­fac­ing with a non-agen­tial AGI is gen­er­ally awk­ward. You can’t di­rectly ask it to do things, or to find a bet­ter way to com­mu­ni­cate. My pro­posal here is to ask ques­tions like “If there were no AGIs in the world, what’s the like­liest way that a per­son would find a cure for Alzheimer’s?” This type of ques­tion does not re­quire the AGI to think through the con­se­quence of its out­put, and it also has other nice prop­er­ties (it should give less weird and alien and hu­man-un­friendly an­swers than the solu­tions a di­rect goal-seek­ing agent would find).

  OK, that’s my grand vi­sion and mo­ti­va­tion, and why I’m hop­ing for “no rea­son­ing about the con­se­quences of one’s out­put what­so­ever”, as op­posed to find­ing self-fulfilling pre­dic­tions. (Maybe very very mild op­ti­miza­tion for the con­se­quences of one’s out­puts is OK, but I’m ner­vous.)

  Your other ques­tion was: if a sys­tem is mak­ing ma­nipu­la­tive pre­dic­tions, to­wards what goal is it ma­nipu­lat­ing? Well, you no­ticed cor­rectly, I’m not sure, and I keep chang­ing my mind. And it may also be differ­ent an­swers de­pend­ing on the al­gorithm de­tails.

  • My top ex­pec­ta­tion is that it will ma­nipu­late to­wards get­ting fur­ther in­puts that its model thinks are typ­i­cal, high-prob­a­bil­ity in­puts. If X im­plies Y, and P(Y) is low, that might some­times spu­ri­ously push down P(X), and thus the sys­tem will pick those X’s that re­sult in high P(Y).

  • My sec­ondary ex­pec­ta­tion is that it might ma­nipu­late to­wards un­am­bigu­ous, low-en­tropy out­puts. This is the ex­pec­ta­tion if the sys­tem picks out the sin­gle most likely on­go­ing long-term con­text, and out­put a pre­dic­tion con­tin­gent on that. (If in­stead the sys­tem ran­domly draws from the prob­a­bil­ity dis­tri­bu­tion of all pos­si­ble con­texts, this wouldn’t hap­pen, as sug­gested by in­ter­stice’s com­ments on this page.) So if X1 leads to one of 500 slightly differ­ent Y1′s (Y1a, Y1b,...), while X2 definitely leads to only one spe­cific Y2, then Y2 is prob­a­bly the most likely sin­gle Y, even if all the Y1′s in ag­gre­gate are like­lier than Y2; so X2 is at an un­fair ad­van­tage.

  • Beyond those two, I sus­pect there can be other goals but they de­pend on the al­gorithm and its heuris­tics.

• steve2152 3 Sep 2019 20:01 UTC

  3 points

  in reply to: John_Maxwell's comment on: Self-su­per­vised learn­ing & ma­nipu­la­tive predictions

  OK, hmm, let me try again then. This would be the sec­tion of the post en­ti­tled “A self-su­per­vised learn­ing al­gorithm in an in­ter­ac­tive en­vi­ron­ment can be­come a ma­nipu­la­tive goal-seeker”.

  I’ve been as­sum­ing all along that the ob­jec­tive func­tion only re­wards the next word. Un­for­tu­nately, it seems that the way to achieve this ob­jec­tive in prac­tice is to search for higher-level longer-term con­texts that sur­round the next word, like when we’re watch­ing TV and we think, “A com­mer­cial break is start­ing.” Know­ing that a com­mer­cial break is start­ing is es­sen­tial for pre­dict­ing the very next frame on the TV screen, but it is also in­ci­den­tally a (im­plicit) pre­dic­tion about what will ap­pear on the screen for the next few min­utes. In other words, you could say that mak­ing ac­cu­rate (pos­si­bly im­plicit) prob­a­bil­is­tic pre­dic­tions about the next many words is in­stru­men­tally use­ful for mak­ing ac­cu­rate prob­a­bil­is­tic pre­dic­tions about the next one word, and is thus re­warded by the ob­jec­tive func­tion. I ex­pect that sys­tems that work well will have to be de­signed this way (i.e. find­ing “con­texts” that en­tail im­plicit pre­dic­tions about many fu­ture words, as a step to­wards pick­ing the sin­gle next word). I think this kind of thing is nec­es­sary to im­ple­ment even very ba­sic things like ob­ject per­ma­nence.

  Then the next step is to sup­pose that the sys­tem (be­ing highly in­tel­li­gent) comes to be­lieve that the pre­dic­tion X will cause other as­pects of the longer-term con­text to be Y. (See the “Hy­poth­e­sis 1” vs “Hy­poth­e­sis 2″ ex­am­ples in the post.) If the sys­tem was pre­vi­ously think­ing that P(X) is high and P(Y) is low, then ideally, the re­al­iza­tion that X im­plies Y will cause the sys­tem to raise P(Y), while keep­ing P(X) at its pre­vi­ous value. This is, af­ter all, the log­i­cally cor­rect up­date, based on the di­rec­tion of causal­ity!

  But if the sys­tem screws up, and low­ers P(X) in­stead of rais­ing P(Y), then it will make a ma­nipu­la­tive pre­dic­tion—the out­put is be­ing cho­sen par­tially for its down­stream in­ter­ac­tive effects. (Not all ma­nipu­la­tive pre­dic­tions are dan­ger­ous, and there might be limits to how strongly it op­ti­mizes its out­puts for their down­stream effects, but I sus­pect that this par­tic­u­lar case can in­deed lead to catas­trophic out­comes, just like we gener­i­cally ex­pect from AIs with real-world hu­man-mis­al­igned goals.)

  Why should the sys­tem screw up this way? Just be­cause the sys­tem’s causal mod­els will some­times have mis­takes, and some­times have un­cer­tain­ties or blank spaces (statis­ti­cal-reg­u­lar­i­ties-of-un­known-cause), and also be­cause hu­mans make this type of mis­take all the time (“One man’s modus po­nens is an­other man’s modus tol­lens”). I sus­pect it will make the right up­date more of­ten than chance, I just don’t see how we can guaran­tee that it will never make the wrong up­date in the ma­nipu­la­tive Y-->X di­rec­tion.

  Does that help?

• steve2152 2 Sep 2019 19:13 UTC

  1 point

  in reply to: John_Maxwell's comment on: Self-su­per­vised learn­ing & ma­nipu­la­tive predictions

  Well, strat­egy 1 is “Keep it from think­ing that it’s in an in­ter­ac­tive en­vi­ron­ment”. Things like “don’t ad­just the weights of the net­work while we ask ques­tions” is a way to pre­vent it from think­ing that it’s in an in­ter­ac­tive en­vi­ron­ment based on first-hand ex­pe­rience—we’re en­g­ineer­ing the ex­pe­rience to not leave traces in its knowl­edge. But to suc­ceed in strat­egy 1, we also need to make sure that it doesn’t come to be­lieve it’s in an in­ter­ac­tive en­vi­ron­ment by other means be­sides first-hand ex­pe­rience, namely by ab­stract rea­son­ing. More de­tails in this com­ment, but ba­si­cally an AGI with in­tro­spec­tive in­for­ma­tion and world-knowl­edge will nat­u­rally over time figure out that it’s an AGI, and to figure out the sorts of en­vi­ron­ments that AGIs are typ­i­cally in, and thus to hy­poth­e­size the ex­is­tence of in­ter­ac­tions even if those in­ter­ac­tions have never hap­pened be­fore, and were not in­tended by the de­signer (e.g. the “Help I’m trapped in a GPU!” type in­ter­ac­tions).

• steve2152 1 Sep 2019 0:28 UTC

  1 point

  in reply to: John_Maxwell's comment on: Self-su­per­vised learn­ing & ma­nipu­la­tive predictions

  Yeah, I think some­thing like that would prob­a­bly work for 1B, but 1B is the easy part. It’s 1C & 1D that are keep­ing me up at night...

• steve2152 31 Aug 2019 0:29 UTC

  2 points

  in reply to: ryan_b's comment on: [Link] Book Re­view: Refram­ing Su­per­in­tel­li­gence (SSC)

  Re­gard­ing “com­pu­ta­tional thresh­old”, my work­ing as­sump­tion is that any given ca­pa­bil­ity X is ei­ther (1) always and for­ever out of reach of a sys­tem by de­sign, or (2) com­pletely use­less, or (3) very likely to be learned by a sys­tem, if the sys­tem has long-term real-world goals. Maybe it takes some com­pu­ta­tional time and effort to learn it, but AIs are not lazy (un­less we pro­gram them to be). AIs are just sys­tems that make good de­ci­sions in pur­suit of a goal, and if “ac­quiring ca­pa­bil­ity X” is in­stru­men­tally helpful to­wards achiev­ing goals in the world, it will prob­a­bly make that de­ci­sion if it can (cf. “In­stru­men­tal con­ver­gence”).

  If I have a life goal that is best ac­com­plished by learn­ing to use a fork­lift, I’ll learn to use a fork­lift, right? Maybe I won’t be very fluid at it, but fine, I’ll op­er­ate it more slowly and de­liber­ately, or de­sign a fork­lift au­topi­lot sub­sys­tem, or what­ever...

• steve2152 30 Aug 2019 15:08 UTC

  2 points

  in reply to: ryan_b's comment on: [Link] Book Re­view: Refram­ing Su­per­in­tel­li­gence (SSC)

  Right, I was us­ing “out­put” in a broad sense of “any way that the sys­tem can causally im­pact the rest of the world”. We can di­vide that into “in­tended out­put chan­nels” (text on a screen etc.) and “un­in­tended out­put chan­nels” (send­ing out ra­dio sig­nals us­ing RAM etc.). I’m fa­mil­iar with a small amount of work on avoid­ing un­in­tended out­put chan­nels (e.g. us­ing ho­mo­mor­phic en­cryp­tion or fancy vac­uum-sealed Fara­day cage boxes).

  Usu­ally the as­sump­tion is that a su­per­in­tel­li­gent AI will figure out what it is, and where it is, and how it works, and what all its out­put chan­nels are (both in­tended and un­in­tended), un­less there is some strong rea­son to be­lieve oth­er­wise (ex­am­ple). I’m not sure this an­swers your ques­tion … I’m a bit con­fused at what you’re get­ting at.

• steve2152 30 Aug 2019 10:45 UTC

  2 points

  in reply to: ryan_b's comment on: [Link] Book Re­view: Refram­ing Su­per­in­tel­li­gence (SSC)

  AGIs will have a causal model of the world. If their own out­put is part of that model, and they work for­ward from there to the real-world con­se­quences of their out­puts, and they choose out­puts partly based on those con­se­quences, then it’s an agent by (my) defi­ni­tion. The out­puts are called “ac­tions” and the con­se­quences are called “goals”. In all other cases, then I’d call it a ser­vice, un­less I’m for­get­ting about some edge cases.

  A sys­tem whose only out­put is text on a screen can be ei­ther a ser­vice or an agent, de­pend­ing on the com­pu­ta­tional pro­cess gen­er­at­ing the text. A sim­ple test is that if there’s a weird, non-ob­vi­ous way to ma­nipu­late the peo­ple read­ing the text (ac­cord­ing the ev­ery­day, bad-con­no­ta­tion sense of “ma­nipu­late”), would the sys­tem take ad­van­tage of it? Agents would do so (by de­fault, un­less they had a com­pli­cated goal in­volv­ing ethics etc.), ser­vices would not by de­fault.

  No­body knows how to build a use­ful AI ca­pa­ble of world-mod­el­ing and for­mu­lat­ing in­tel­li­gent plans but which is not an agent, al­though I’m per­son­ally hope­ful that it might be pos­si­ble by self-su­per­vised learn­ing (cf. Self-Su­per­vised Learn­ing and AGI safety).

• steve2152 29 Aug 2019 14:26 UTC

  6 points

  on: If the “one cor­ti­cal al­gorithm” hy­poth­e­sis is true, how should we up­date about timelines and take­off speed?

  My own up­dates af­ter I wrote that were:

  • In­creased like­li­hood of self-su­per­vised learn­ing al­gorithms as ei­ther a big part or even the en­tirety of the tech­ni­cal path to AGI—in­so­far as self-su­per­vised learn­ing is the lion’s share of how the neo­cor­tex learn­ing al­gorithm sup­pos­edly works. That’s why I’ve been writ­ing posts like Self-Su­per­vised Learn­ing and AGI safety.

  • Shorter timelines and faster take­off, in­so­far as we think the al­gorithm is not over­whelm­ingly complicated

  • In­creased like­li­hood of “one al­gorithm to rule them all” over Com­pre­hen­sive AI Ser­vices. This might be on the meta-level of one learn­ing al­gorithm to rule them all, and we feed it biol­ogy books to get a su­per­in­tel­li­gent biol­o­gist, and sep­a­rately we feed it psy­chol­ogy books and non­fic­tion TV to get a su­per­in­tel­li­gent psy­cholog­i­cal charis­matic ma­nipu­la­tor, etc. Or it might be on the base level of one trained model to rule them all, and we train it with all 50 mil­lion books and 100,000 years of YouTube and any­thing else we can find. The lat­ter can ul­ti­mately be more ca­pa­ble (you un­der­stand biol­ogy pa­pers bet­ter if you also un­der­stand statis­tics, etc. etc.), but on the other hand the former is more likely if there are scal­ing limits where mem­ory ac­cess grinds to a halt af­ter too many gi­gabytes get loaded into the world-model, or things like that. Either way, it would make it like­lier for AGI (or at least the fi­nal miss­ing in­gre­di­ent of AGI) to be de­vel­oped in one place, i.e. the search-en­g­ine model rather than the open-source soft­ware model.

• steve2152 27 Aug 2019 10:46 UTC

  1 point

  on: Embed­ded Agency via Abstraction

  Peo­ple (and robots) model the world by start­ing with sen­sor data (vi­sion, pro­pri­o­cep­tion, etc.), then find­ing low-level (spa­tiotem­po­rally-lo­cal­ized) pat­terns in that data, then higher-level pat­terns in the pat­terns, pat­terns in the pat­terns in the pat­terns, etc. I’m try­ing to un­der­stand how this re­lates to “ab­strac­tion” as you’re talk­ing about it.

  Sen­sor data, say the bits recorded by a video cam­era, is not a causal di­a­gram, but it is already an “ab­strac­tion” in the sense that it has mu­tual in­for­ma­tion with the part of the world it’s look­ing at, but is many or­ders of mag­ni­tude less com­pli­cated. Do you see a video cam­era as an ab­strac­tion-cre­ator /​ map-maker by it­self?

  What if the video cam­era has a MPEG con­verter? MPEGs can (I think) rec­og­nize that low-level pat­tern X tends to fol­low low-level pat­tern Y, and this is more-or-less the same low-level prim­i­tive out of which which hu­mans build their so­phis­ti­cated causal un­der­stand­ing of the world (ac­cord­ing to my cur­rent un­der­stand­ing of the hu­man brain’s world-mod­el­ing al­gorithms). So is a video cam­era with MPEG con­verter an ab­strac­tion-cre­ator /​ map-maker? What’s your think­ing?

• steve2152 26 Aug 2019 9:07 UTC

  2 points

  on: Op­ti­miza­tion Provenance

  Cou­ple things:

  (1) You might give some thought to try­ing to copy (or at least un­der­stand) the world model frame­work of the hu­man brain. There’s un­cer­tainty in how that works, but a lot is known, and you’ll at least be work­ing to­wards some­thing that we know for sure is ca­pa­ble of get­ting built up to a hu­man level world-model within a rea­son­able amount of time and com­pu­ta­tion. As best as I can tell (and I’m work­ing hard to un­der­stand it my­self), and grossly over­sim­plify­ing, it’s a data struc­ture with billions of dis­crete con­cepts, and trans­for­ma­tions be­tween those con­cepts (com­po­si­tion, cause-effect, anal­ogy, etc...prob­a­bly all of those are built out of the same ba­sic “trans­for­ma­tion ma­chin­ery” with differ­ent con­texts act­ing as meta­data). All these con­cepts are sit­ting in the top layer of some kind of loose hi­er­ar­chy, whose low­est layer con­sists of (higher-level-con­text-de­pen­dent) prob­a­bil­ity dis­tri­bu­tions over spa­tiotem­po­ral se­quences of sen­sory in­puts. See my Jeff Hawk­ins post for one pos­si­ble point of de­par­ture. I’ve found a cou­ple other refer­ences that are in­di­rectly helpful, and like I said, I’m still try­ing to figure it out. I’m still try­ing to un­der­stand the “sheaves” ap­proach , so I won’t com­ment on how these com­pare.

  (2) “This con­cep­tion will be the re­sult of an op­ti­mizer, and so this should be in the op­ti­miza­tion prove­nance”—this seems to be im­por­tant and I don’t un­der­stand it. Bet­ter un­der­stand­ing the world con­sists (in part) of chunk­ing se­quences of events and ac­tions, sup­press­ing in­ter­me­di­ate steps. Thus we say and think “I’ll put some milk in my coffee,” leav­ing out the steps like un­screw­ing the top of the jug. The pro­cess of “ex­plore the world model, chunk­ing se­quences of events when ap­pro­pri­ate” is (I sus­pect) es­sen­tial to mak­ing the world-model us­able and pow­er­ful, and needs to be re­peated mil­lions of times in ev­ery nook and cranny of the world model, and thus this is a pro­cess that an over­seer would have lit­tle choice but to ap­prove in gen­eral, I think. But this pro­cess can find and chunk ma­nipu­la­tive causal path­ways just as well as any other kind of path­way. And once ma­nipu­la­tion is pack­aged up in­side a chunk, you won’t need op­ti­miza­tion per se to ma­nipu­late, it will just be an ob­vi­ous step in the pro­cess of do­ing some­thing, just like un­screw­ing the top of the jug is an ob­vi­ous step in putting-milk-into-coffee. I’m not sure how you pro­pose to stop that from hap­pen­ing.

• steve2152 25 Aug 2019 11:46 UTC

  1 point

  in reply to: Gurkenglas's comment on: Does Agent-like Be­hav­ior Im­ply Agent-like Ar­chi­tec­ture?

  No, I would try to rule out stars based on “a-pri­ori-speci­fi­able con­se­quence”—say­ing that “stars shine” would be paint­ing a tar­get around the ar­row, i.e. rea­son­ing about what the sys­tem would ac­tu­ally do and then say­ing that the sys­tem is go­ing to do that. For ex­am­ple, ex­pect­ing bac­te­ria to take ac­tions that max­i­mize in­clu­sive ge­netic fit­ness would cer­tainly qual­ify as “a pri­ori speci­fi­able”. The other part is “more likely than chance”, which I sup­pose en­tails a range of pos­si­ble ac­tions/​be­hav­iors, with differ­ent ac­tions/​be­hav­iors in­voked in differ­ent pos­si­ble uni­verses, but lead­ing to the same con­se­quence re­gard­less. (You can see how ev­ery step I make to­wards be­ing spe­cific here is also a step to­wards mak­ing my “the­o­rem” com­pletely triv­ial, X=X.)

• steve2152 25 Aug 2019 1:17 UTC

  3 points

  on: Does Agent-like Be­hav­ior Im­ply Agent-like Ar­chi­tec­ture?

  Let’s say “agent-like be­hav­ior” is “tak­ing ac­tions that are more-likely-than-chance to cre­ate an a-pri­ori-speci­fi­able con­se­quence” (this defi­ni­tion in­cludes bac­te­ria).

  Then I’d say this re­quires “agent-like pro­cesses”, in­volv­ing (at least) all 4 of: (1) hav­ing ac­cess to some in­for­ma­tion about the world (at least the lo­cal en­vi­ron­ment), in­clud­ing in par­tic­u­lar (2) how one’s ac­tions af­fect the world. This in­for­ma­tion can come ei­ther baked into the de­sign (bac­te­ria, gi­ant lookup table), and/​or from pre­vi­ous ex­pe­rience (RL), and/​or via rea­son­ing from in­put data. It also needs (3) an abil­ity to use this in­for­ma­tion to choose ac­tions that are like­lier-than-chance to achieve the con­se­quence in ques­tion (again, the out­come of this search pro­cess could be baked into the de­sign like bac­te­ria, or it could be calcu­lated on-the-fly like hu­man fore­sight), and of course (4) a ten­dency to ac­tu­ally ex­e­cute those ac­tions in ques­tion.

  I feel like this is al­most triv­ial, like I’m just restat­ing the same thing in two differ­ent ways… I mean, if there’s no mu­tual in­for­ma­tion be­tween the agent and the world, its ac­tions can only be effec­tive only in­so­far as the ex­act same ac­tion would be effec­tive when ex­e­cuted in a ran­dom lo­ca­tion of a ran­dom uni­verse. (Does con­tract­ing your own mus­cle count as “ac­com­plish­ing some­thing with­out any world knowl­edge”?)

  Any­way, where I’m re­ally skep­ti­cal here is in the term “ar­chi­tec­ture”. “Ar­chi­tec­ture” in ev­ery­day us­age usu­ally im­plies soft­ware prop­er­ties that are ob­vi­ous parts of how a pro­gram is built, and prob­a­bly put in on pur­pose. (Is there a more spe­cific defi­ni­tion of “ar­chi­tec­ture” you had in mind?) I’m pretty doubt­ful that the in­gre­di­ents 1-4 have to be part of the “ar­chi­tec­ture” in that sense. For ex­am­ple, I’ve been think­ing a lot about self-su­per­vised learn­ing al­gorithms, which have in­gre­di­ent (1) by de­sign and have (3) sorta in­ci­den­tally. The other two in­gre­di­ents (2) and (4) are definitely not part of the “ar­chi­tec­ture” (in the sense above). But I’ve ar­gued that they can both oc­cur as un­in­tended side-effects of its op­er­a­tion: See here, and also here for more de­tails about (2). And thus I ar­gue at that first link that this sys­tem can have agent-like be­hav­ior.

  (And what’s the “ar­chi­tec­ture” of a bac­te­ria any­way? Not a rhetor­i­cal ques­tion.)

  Sorry if this is all in­cor­rect and/​or not in the spirit of your ques­tion.

• steve2152 23 Aug 2019 11:38 UTC

  1 point

  in reply to: interstice's comment on: Self-su­per­vised learn­ing & ma­nipu­la­tive predictions

  Thanks, that’s helpful! I’ll have to think about the “self-con­sis­tent prob­a­bil­ity dis­tri­bu­tion” is­sue more, and thanks for the links. (ETA: Mean­while I also added an “Up­date 2″ to the post, offer­ing a differ­ent way to think about this, which might or might not be helpful.)

  Let me try the gra­di­ent de­scent ar­gu­ment again (and note that I am sym­pa­thetic, and in­deed I made (what I think is) that ex­act ar­gu­ment a few weeks ago, cf. Self-Su­per­vised Learn­ing and AGI Safety, sec­tion ti­tle “Why won’t it try to get more pre­dictable data?”). My ar­gu­ment here is not as­sum­ing there’s a policy of try­ing to get more pre­dictable data for its own sake, but rather that this kind of be­hav­ior arises as a side-effect of an al­gorith­mic pro­cess, and that all the in­gre­di­ents of that pro­cess are ei­ther things we would pro­gram into the al­gorithm our­selves or things that would be in­cen­tivized by gra­di­ent de­scent.

  The in­gre­di­ents are things like “Look for and learn pat­terns in all ac­cessible data”, which in­cludes both low-level pat­terns in the raw data, higher-level pat­terns in the lower-level pat­terns, and (per­haps un­in­ten­tion­ally) pat­terns in ac­cessible in­for­ma­tion about its own thought pro­cess (“After I vi­su­al­ize the shape of an elephant tusk, I of­ten vi­su­al­ize an elephant shortly there­after”). It in­cludes search­ing for trans­for­ma­tions (cause-effect, com­po­si­tion, analo­gies, etc.) be­tween any two pat­terns it already knows about (“sneak­ers are a type of shoe”, or more prob­le­mat­i­cally, “my thought pro­cesses re­sem­ble the as­so­ci­a­tive mem­ory of an AGI”), and cat­a­loging these trans­for­ma­tions when they’re found. Stuff like that.

  So, “make smart hy­pothe­ses about one’s own em­bod­ied situ­a­tion” is definitely an un­in­tended side-effect, and not re­warded by gra­di­ent de­scent as such. But as its world-model be­comes more com­pre­hen­sive, and as it con­tinues to au­to­mat­i­cally search for pat­terns in what­ever in­for­ma­tion it has ac­cess to, “make smart hy­pothe­ses about one’s own em­bod­ied situ­a­tion” would just be some­thing that hap­pens nat­u­rally, un­less we some­how pre­vent it (and I can’t see how to pre­vent it). Like­wise, “model one’s own real-world causal effects on down­stream data” is nei­ther de­sired by us nor re­warded (as such) by gra­di­ent de­scent. But it can hap­pen any­way, as a side-effect of the usu­ally-lo­cally-helpful rule of “search through the world-model for any pat­terns and re­la­tion­ships which may im­pact our be­liefs about the up­com­ing data”. Like­wise, we have the gen­er­ally-helpful rule “Hy­poth­e­size pos­si­ble higher-level con­texts that span an ex­tended swathe of text sur­round­ing the next word to be pre­dicted, and pick one such con­text based on how sur­pris­ing it would be based on what it knows about the pre­ced­ing text and the world-model, and then make a pre­dic­tion con­di­tional on that con­text”. All these in­gre­di­ents com­bine to get the patholog­i­cal be­hav­ior of choos­ing “Help I’m trapped in a GPU”. That’s my ar­gu­ment, any­way...

• steve2152 22 Aug 2019 10:04 UTC

  1 point

  in reply to: Joar Skalse's comment on: Two senses of “op­ti­mizer”

  RE “make the su­per­in­tel­li­gence as­sume that it is dis­em­bod­ied”—I’ve been think­ing about this a lot re­cently (see The Self-Unaware AI Or­a­cle) and agree with Viliam that knowl­edge-of-one’s-em­bod­i­ment should be the de­fault as­sump­tion. My rea­son­ing is: A good world-mod­el­ing AI should be able to rec­og­nize pat­terns and build con­cep­tual trans­for­ma­tions be­tween any two things it knows about, and also should be able to do rea­son­ing over ex­tended pe­ri­ods of time. OK, so let’s say it’s try­ing to figure some­thing out some­thing about biol­ogy, and it vi­su­al­izes the shape of a tree. Now it (by de­fault) has the in­tro­spec­tive in­for­ma­tion “A tree has just ap­peared in my imag­i­na­tion!”. Like­wise, if it goes through any kind of rea­son­ing pro­cess, and can self-re­flect on that rea­son­ing pro­cess, then it can learn (via the same pat­tern-rec­og­niz­ing al­gorithm it uses for the ex­ter­nal world) how that rea­son­ing pro­cess works, like “I seem to have some kind of as­so­ci­a­tive mem­ory, I seem to have a ca­pac­ity for build­ing hi­er­ar­chi­cal gen­er­a­tive mod­els, etc.” Then it can rec­og­nize that these are the same in­gre­di­ents pre­sent in those AGIs it read about in the news­pa­per. It also knows a higher-level pat­tern “When two things are built the same way, maybe they’re of the same type.” So now it has a hy­poth­e­sis that it’s an AGI run­ning on a com­puter.

  It may be pos­si­ble to pre­vent this cas­cade of events, by some­how mak­ing sure that “I am imag­in­ing a tree” and similar things never get writ­ten into the world model. I have this vi­sion of two data-types, “in­tro­spec­tive in­for­ma­tion” and “world-model in­for­ma­tion”, and your static type-checker en­sures that the two never co-min­gle. And voila, AI Safety! That would be awe­some. I hope some­body figures out how to do that, be­cause I sure haven’t. (Ad­mit­tedly, I have nei­ther time nor rele­vant back­ground knowl­edge to try prop­erly.) I’m also slightly con­cerned that, even if you figure out a way to cut off in­tro­spec­tive knowl­edge, it might in­ci­den­tally pre­vent the sys­tem from do­ing good rea­son­ing, but I cur­rently lean op­ti­mistic on that.

• steve2152 22 Aug 2019 1:30 UTC

  4 points

  on: Two senses of “op­ti­mizer”

  I think I have an ex­am­ple of “an op­ti­mizer_1 could turn into an op­ti­mizer_2 un­ex­pect­edly if it be­comes suffi­ciently pow­er­ful”. I posted it a cou­ple days ago: Self-su­per­vised learn­ing & ma­nipu­la­tive pre­dic­tions. A self-su­per­vised learn­ing sys­tem is an op­ti­mizer_1: It’s try­ing to pre­dict masked bits in a fixed, pre-loaded set of data. This task does not en­tail in­ter­act­ing with the world, and we would pre­sum­ably try hard to de­sign it not to in­ter­act with the world.

  How­ever, if it was a pow­er­ful learn­ing sys­tem with world-knowl­edge (via its in­put data) and in­tro­spec­tive ca­pa­bil­ities, it would even­tu­ally figure out that it’s an AGI and might hy­poth­e­size what en­vi­ron­ment it’s in, and then hy­poth­e­size that its op­er­a­tions could af­fect its data stream via un­in­tended causal path­ways, e.g. send­ing out ra­dio sig­nals. Then, if it used cer­tain plau­si­ble types of heuris­tics as the ba­sis for its pre­dic­tions of masked bits, it could wind up mak­ing choices based on their down­stream effects on it­self via ma­nipu­lat­ing the en­vi­ron­ment. In other words, it starts act­ing like an op­ti­mizer_2.

  I’m not su­per-con­fi­dent about any of this and am open to crit­i­cism. (And I agree with you that this a use­ful dis­tinc­tion re­gard­less; in­deed I was ar­gu­ing a similar (but weaker) point re­cently, maybe not as el­e­gantly, at this link)

• steve2152 21 Aug 2019 14:13 UTC

  3 points

  in reply to: Charlie Steiner's comment on: Self-Su­per­vised Learn­ing and AGI Safety

  Thanks, that’s helpful!

  The way I’m cur­rently think­ing about it, if we have an or­a­cle that gives su­per­in­tel­li­gent and non-ma­nipu­la­tive an­swers, things are look­ing pretty good for the fu­ture. When you ask it to de­sign a new drug, you also ask some fol­low-up ques­tions like “How does the drug work?” and “If we de­ploy this solu­tion, how might this im­pact the life of a typ­i­cal per­son in 20 years time?” Maybe it won’t always be able to give great an­swers, but as long as it’s not try­ing to be ma­nipu­la­tive, it seems like we ought to be able to use such a sys­tem safely. (This would, in­ci­den­tally, en­tail not let­ting idiots use the sys­tem.)

  I agree that ex­tract­ing in­for­ma­tion from a self-su­per­vised learner is a hard and open prob­lem. I don’t see any rea­son to think it’s im­pos­si­ble. The two gen­eral ap­proaches would be:

  1. Ma­nipu­late the self-su­per­vised learn­ing en­vi­ron­ment some­how. Ba­si­cally, the sys­tem is go­ing to know lots of differ­ent high-level con­texts in which the statis­tics of low-level pre­dic­tions are differ­ent—think about how GPT-2 can imi­tate both mid­dle school es­says and fan-fic­tion. We would need to teach it a con­text in which we ex­pect the text to re­flect profound truths about the world, be­yond what any hu­man knows. That’s tricky be­cause we don’t have any such texts in our database. But maybe if we put a spe­cial to­ken in the 50 most clear and in­sight­ful jour­nal ar­ti­cles ever writ­ten, and then stick that same to­ken in our ques­tion prompt, then we’ll get bet­ter an­swers. That’s just an ex­am­ple, maybe there are other ways.

  2. For­get about text pre­dic­tion, and build an en­tirely sep­a­rate in­put-out­put in­ter­face into the world model. The world model (if it’s vaguely brain-like) is “just” a data struc­ture with billions of dis­crete con­cepts, and trans­for­ma­tions be­tween those con­cepts (com­po­si­tion, cause-effect, anal­ogy, etc...prob­a­bly all of those are built out of the same ba­sic “trans­for­ma­tion ma­chin­ery”). All these con­cepts are sit­ting in the top layer of some kind of hi­er­ar­chy, whose low­est layer con­sists of prob­a­bil­ity dis­tri­bu­tions over short snip­pets of text (for a lan­guage model, or more gen­er­ally what­ever the in­put is). So that’s the world model data struc­ture. I have no idea how to build a new in­ter­face into this data struc­ture, or what that in­ter­face would look like. But I can’t see why that should be im­pos­si­ble...

• steve2152 21 Aug 2019 13:38 UTC

  1 point

  in reply to: G Gordon Worley III's comment on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  I do think I un­der­stand that. I see E as a means to an end. It’s a way to rank-or­der choices and thus make good choices. If I ap­ply an af­fine trans­for­ma­tion to E, e.g. I’m way too op­ti­mistic about ab­solutely ev­ery­thing in a com­pletely uniform way, then I still make the same choice, and the choice is what mat­ters. I just want my AGI to do the right thing.

  Here, I’ll try to put what I’m think­ing more starkly. Let’s say I some­how de­sign a com­par­a­tive AGI. This is a sys­tem which can take a merit func­tion U, and two choices C_A and C_B, and it can pre­dict which of the two choices C_A or C_B would be bet­ter ac­cord­ing to merit func­tion U, but it has no idea how good ei­ther of those two choices ac­tu­ally are on any ab­solute scale. It doesn’t know whether C_A is won­der­ful while C_B is even bet­ter, or whether C_A is awful while C_B is merely so-so, both of those just re­turn the same an­swer, “C_B is bet­ter”. As­sume it’s not om­ni­scient, so its com­par­i­sons are not always cor­rect, but that it’s still im­pres­sively su­per­in­tel­li­gent.

  A com­par­a­tive AGI does not suffer the op­ti­mizer’s curse, right? It never forms any be­liefs about how good its choices will turn out, so it couldn’t pos­si­bly be sys­tem­at­i­cally dis­ap­pointed. There’s always noise and un­cer­tainty, so there will be times when its sec­ond-high­est-ranked choice would ac­tu­ally turn out bet­ter than its high­est-ranked choice. But that hap­pens less of­ten than chance. There’s no sys­tem­atic prob­lem: in ex­pec­ta­tion, the best thing to do (as mea­sure by U) is always to take its top-ranked choice.

  Now, it seems to me that, if I go to the AGIs-R-Us store, and I see a nor­mal AGI and a com­par­a­tive AGI side-by-side on the shelf, I would have no strong opinion about which one of them I should buy. If I ask ei­ther one to do some­thing, they’ll take the same se­quence of ac­tions in the same or­der, and get the same re­sult. They’ll in­vest my money in the same stocks, offer me the same ad­vice, etc. etc. In par­tic­u­lar, I would worry about Good­hart’s law (i.e. giv­ing my AGI the wrong func­tion U) with ei­ther of these AGIs to the ex­act same ex­tent and for the ex­act same rea­son...even though one is sub­ject to op­ti­mizer’s curse and the other isn’t.

• steve2152 21 Aug 2019 12:26 UTC

  1 point

  in reply to: G Gordon Worley III's comment on: Good­hart’s Curse and Limi­ta­tions on AI Alignment

  I don’t think it’s re­lated to mild op­ti­miza­tion. Pick a tar­get T that can be ex­ceeded (won­der­ful fu­ture, even if it’s not the ab­solute the­o­ret­i­cally best pos­si­ble fu­ture). Es­ti­mate what choice Cmax is (as far as we can tell) the #1 very best by that met­ric. We ex­pect Cmax to give value E, and it turns out to be V<E, but V is still likely to ex­ceed T, or at least like­lier than any other choice. (In­so­far as that’s not true, it’s Good­hart.) Op­ti­mizer curse, i.e. V<E, does not seem to be a prob­lem, or even rele­vant, be­cause I don’t ul­ti­mately care about E. Maybe the AI doesn’t even tell me what E is. Maybe the AI doesn’t even bother guess­ing what E is, it only calcu­lates that Cmax seems to be bet­ter than any other choice.

• steve2152 21 Aug 2019 10:45 UTC

  1 point

  in reply to: John_Maxwell's comment on: Self-Su­per­vised Learn­ing and AGI Safety

  Ah, thanks for clar­ify­ing.

  The first en­try on my “list of patholog­i­cal things” wound up be­ing a full blog post in length: See Self-su­per­vised learn­ing and ma­nipu­la­tive pre­dic­tions.

  RE dae­mons, I wrote in that post (and have been as­sum­ing all along): “I’m as­sum­ing that we will not do a meta-level search for self-su­per­vised learn­ing al­gorithms… In­stead, I am as­sum­ing that the self-su­per­vised learn­ing al­gorithm is known and fixed (e.g. “Trans­former + gra­di­ent de­scent” or “what­ever the brain does”), and that the pre­dic­tive model it cre­ates has a known frame­work, struc­ture, and mod­ifi­ca­tion rules, and that only its spe­cific con­tents are a hard-to-in­ter­pret com­pli­cated mess.” The con­tents of a world-model, as I imag­ine it, is a big data struc­ture con­sist­ing of gajillions of “con­cepts” and “trans­for­ma­tions be­tween con­cepts”. It’s a pas­sive data struc­ture, there­fore not a “dae­mon” in the usual sense. Then there’s a KANSI (Known Al­gorithm Non Self Im­prov­ing) sys­tem that’s ac­cess­ing and edit­ing the world model. I also wouldn’t call that a “dae­mon”, in­stead I would say “This al­gorithm we wrote can have patholog­i­cal be­hav­ior...”