Daniel Kokotajlo

• Daniel Kokotajlo 19 Oct 2019 20:29 UTC

  9 points

  in reply to: CellBioGuy's comment on: Su­per­in­tel­li­gence 7: De­ci­sive strate­gic advantage

  I think you are look­ing at this wrong. Yes, they had help from lo­cal re­bel­lions and mal­con­tents. So would an AGI. An AGI tak­ing over the world wouldn’t nec­es­sar­ily look like robots vs. hu­mans; it might look like the out­break of World War 3 be­tween var­i­ous hu­man fac­tions, ex­cept that the AGI was ma­nipu­lat­ing things be­hind the scenes and/​or act­ing as a “strate­gic ad­vi­sor” to one of the fac­tions. And when the dust set­tles, some­how the AGI is in charge...

  So yeah, I think it re­ally is fair to say that the Span­ish man­aged to con­quer em­pires of mil­lions with just a few hun­dred men. Twice.

• Daniel Kokotajlo 17 Oct 2019 19:10 UTC

  2 points

  in reply to: riceissa's comment on: Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

  Hmm, I like that. I won­der what A&M would say in re­sponse. And I agree this is an im­por­tant and rele­vant differ­ence be­tween the case of prefer­ences and the case of sci­ence.

  I still don’t think A&M show that the sim­plest ex­pla­na­tion is a de­gen­er­ate de­com­po­si­tion. They show that if it is, then Oc­cam’s Ra­zor won’t be suffi­cient, and more­over that there are some de­gen­er­ate de­com­po­si­tions pretty close to max­i­mally sim­ple. But they don’t do much to rule out the pos­si­bil­ity that the sim­plest ex­pla­na­tion is the in­tended one.

• Daniel Kokotajlo 15 Oct 2019 15:27 UTC

  2 points

  in reply to: bgold's comment on: Daniel Koko­ta­jlo’s Shortform

  I agree. I don’t think agents will out­com­pete tools in ev­ery do­main; in­deed in most do­mains per­haps spe­cial­ized tools will even­tu­ally win (already, we see hu­mans be­ing re­placed by ex­pen­sive spe­cial­ized ma­chin­ery, or ex­pen­sive hu­man spe­cial­ists, lots of places). But I still think that there will be strong com­pet­i­tive pres­sure to cre­ate agent AGI, be­cause there are many im­por­tant do­mains where agency is an ad­van­tage.

• Daniel Kokotajlo 10 Oct 2019 16:27 UTC

  2 points

  in reply to: Pattern's comment on: Daniel Koko­ta­jlo’s Shortform

  The ul­ti­mate test will be see­ing whether the pre­dic­tions it makes come true—whether agenty mesa-op­ti­miz­ers arise of­ten, whether hu­mans with tools get out­com­peted by agent AGI.

  In the mean­time, it’s not too hard to look for con­firm­ing or dis­con­firm­ing ev­i­dence. For ex­am­ple, the fact that mil­i­taries and cor­po­ra­tions that make a plan and then task their sub­or­di­nates with strictly fol­low­ing the plan in­vari­ably do worse than those who make a plan and then give their sub­or­di­nates ini­ti­a­tive and flex­i­bil­ity to learn and adapt on the fly… seems like con­firm­ing ev­i­dence. (See: ag­ile de­vel­op­ment model, the im­por­tance of iter­a­tion and feed­back loops in startup cul­ture, etc.) Whereas per­haps the fact that AlphaZero is so good de­spite lack­ing a learn­ing mod­ule is dis­con­firm­ing ev­i­dence.

  As for a test, well we’d need to have some­thing that pro­po­nents and op­po­nents agree to dis­agree on, and that might be hard to find. Most tests I can think of now don’t work be­cause ev­ery­one would agree on what the prob­a­ble out­come is. I think the best I can do is: Some­day soon we might be able to test an agenty ar­chi­tec­ture and a non-agenty ar­chi­tec­ture in some big com­plex novel game en­vi­ron­ment, and this con­jec­ture would pre­dict that for suffi­ciently com­plex and novel en­vi­ron­ments the agenty ar­chi­tec­ture would win.

• Daniel Kokotajlo 9 Oct 2019 22:45 UTC

  LW: 1 AF: 1

  AF

  in reply to: rohinmshah's comment on: Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

  I feel like there’s a big differ­ence be­tween “similar com­plex­ity” and “the same com­plex­ity.” Like, if we have the­ory T and then we have the­ory T* which adds some sim­ple un­ob­tru­sive twist to it, we get an­other the­ory which is of similar com­plex­ity… yet re­al­is­ti­cally an Oc­cam’s-Ra­zor-driven search pro­cess is not go­ing to set­tle on T*, be­cause you only get T* by mod­ify­ing T. And if I’m wrong about this then it seems like Oc­cam’s Ra­zor is bro­ken in gen­eral; in any do­main there are go­ing to be ways to turn T’s into T*’s. But Oc­cam’s Ra­zor is not bro­ken in gen­eral (I feel).

  Maybe this is the ar­gu­ment you an­ti­ci­pate above with ”...we aren’t ac­tu­ally choos­ing ran­domly.” Oc­cam’s Ra­zor isn’t ran­dom. Again, I might agree with you that in­tu­itively Oc­cam’s Ra­zor seems more use­ful in physics than in prefer­ence-learn­ing. But in­tu­itions are not ar­gu­ments, and any­how they aren’t ar­gu­ments that ap­peared in the text of A&M’s pa­per.

• Daniel Kokotajlo 8 Oct 2019 22:07 UTC

  9 points

  on: Daniel Koko­ta­jlo’s Shortform

  For the past year I’ve been think­ing about the Agent vs. Tool de­bate (e.g. thanks to read­ing CAIS/​Refram­ing Su­per­in­tel­li­gence) and also about em­bed­ded agency and mesa-op­ti­miz­ers and all of these top­ics seem very re­lated now… I keep find­ing my­self at­tracted to the fol­low­ing ar­gu­ment skele­ton:

  Rule 1: If you want any­thing un­usual to hap­pen, you gotta ex­e­cute a good plan.

  Rule 2: If you want a good plan, you gotta have a good plan­ner and a good world-model.

  Rule 3: If you want a good world-model, you gotta have a good learner and good data.

  Rule 4: Hav­ing good data is it­self an un­usual hap­pen­stance, so by Rule 1 if you want good data you gotta ex­e­cute a good plan.

  Put­ting it all to­gether: Agents are things which have good plan­ner and learner ca­pac­i­ties and are hooked up to ac­tu­a­tors in some way. Per­haps they also are “seeded” with a de­cent world-model to start off with. Then, they get a nifty feed­back loop go­ing: They make de­cent plans, which al­low them to get de­cent data, which al­lows them to get bet­ter world-mod­els, which al­lows them to make bet­ter plans and get bet­ter data so they can get great world-mod­els and make great plans and… etc. (The best agents will also be im­prov­ing on their learn­ing and plan­ning al­gorithms! Hu­mans do this, for ex­am­ple.)

  Em­piri­cal con­jec­ture: Tools suck; agents rock, and that’s why. It’s also why agenty mesa-op­ti­miz­ers will arise, and it’s also why hu­mans with tools will even­tu­ally be out­com­peted by agent AGI.

• Daniel Kokotajlo 8 Oct 2019 18:53 UTC

  28 points

  on: Daniel Koko­ta­jlo’s Shortform

  
  

  My baby daugh­ter was born two weeks ago, and in honor of her ex­is­tence I’m build­ing a list of about 100 tech­nol­ogy-re­lated fore­cast­ing ques­tions, which will re­solve in 5, 10, and 20 years. Ques­tions like “By the time my daugh­ter is 5/​10/​20 years old, the av­er­age US cit­i­zen will be able to hail a driver­less taxi in most ma­jor US cities.” (The idea is, ty­ing it to my daugh­ter’s age will make it more fun and also in­crease the like­li­hood that I ac­tu­ally go back and look at it 10 years later.)

  I’d love it if the ques­tions were on­line some­where so other peo­ple could record their an­swers too. Does this seem like a good idea? Hive mind, I be­seech you: Help me spot ways in which this could end badly!

  On a more pos­i­tive note, any sug­ges­tions for how to do it? Any ex­pres­sions of in­ter­est in mak­ing pre­dic­tions with me?

  Thanks!

Daniel Koko­ta­jlo’s Shortform

• Daniel Kokotajlo 8 Oct 2019 13:59 UTC

  5 points

  in reply to: riceissa's comment on: Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

  Thanks! OK, so I agree that nor­mally in do­ing sci­ence we are fine with just pre­dict­ing what will hap­pen, there’s no need to de­com­pose into Laws and Con­di­tions. Whereas with value learn­ing we are try­ing to do more than just pre­dict be­hav­ior; we are try­ing to de­com­pose into Plan­ner and Re­ward so we can max­i­mize Re­ward.

  How­ever the sci­ence case can be made analo­gous in two ways. First, as Eigil says be­low, re­al­is­ti­cally we don’t have ac­cess to ALL be­hav­ior or ALL events, so we will have to ac­cept that the pre­dic­tor which pre­dicted well so far might not pre­dict well in the fu­ture. Thus if Oc­cam’s Ra­zor set­tles on weird de­gen­er­ate pre­dic­tors, it might also set­tle on one that pre­dicts well up un­til time T but then pre­dicts poorly af­ter that.

  Se­cond, (this is the way I went, with coun­ter­fac­tu­als) sci­ence isn’t all about pre­dic­tion. Part of sci­ence is about an­swer­ing coun­ter­fac­tual ques­tions like “what would have hap­pened if...” And typ­i­cally the way to an­swer these ques­tions is by de­com­pos­ing into Laws + Con­di­tions and then do­ing a sur­gi­cal in­ter­ven­tion on the con­di­tions and then ap­ply­ing the same Laws to the new con­di­tions.

  So, for ex­am­ple, if we use Oc­cam’s Ra­zor to find Laws+Con­di­tions for our uni­verse, and some­how it set­tles on the de­gen­er­ate pair “Con­di­tions := null, Laws := se­quence of events E hap­pens” then all our coun­ter­fac­tual queries will give bo­gus an­swers—for ex­am­ple, “what would have hap­pened if we had posted the nu­clear launch codes on the In­ter­net?” An­swer: “Vary­ing the Con­di­tions but hold­ing the Laws fixed… it looks like E would have hap­pened. So yeah, post­ing launch codes on the In­ter­net would have been fine, wouldn’t have changed any­thing.”

• Daniel Kokotajlo 8 Oct 2019 1:41 UTC

  1 point

  in reply to: Pattern's comment on: Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

  I don’t fol­low?

• Daniel Kokotajlo 8 Oct 2019 1:16 UTC

  1 point

  in reply to: steve2152's comment on: Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

  The trick is that you can use the sim­plest method for con­struct­ing E in your state­ment “L=0 and E just hap­pens.” So e.g. if you have some sim­ple Laws l and Con­di­tions c such that l(c) = E, your state­ment can be “L=0 and l(c) just hap­pens.”

• Daniel Kokotajlo 8 Oct 2019 1:06 UTC

  LW: 7 AF: 4

  AF

  in reply to: rohinmshah's comment on: Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

  Thanks! I’m not sure I fol­low you. Here’s what I think you are say­ing:

  --Oc­cam’s Ra­zor will be suffi­cient for pre­dict­ing hu­man be­hav­ior of course; it just isn’t suffi­cient for find­ing the in­tended plan­ner-re­ward pair. Be­cause (A) the sim­plest way to pre­dict hu­man be­hav­ior has noth­ing to do with plan­ners and re­wards, and so (B) the sim­plest plan­ner-re­ward pair will be de­gen­er­ate or weird as A&M ar­gue.

  --You agree that this ar­gu­ment also works for Laws+Ini­tial Con­di­tions; Oc­cam’s Ra­zor is gen­er­ally in­suffi­cient, not just in­suffi­cient for in­fer­ring prefer­ences of ir­ra­tional agents!

  --You think the ar­gu­ment is more likely to work for in­fer­ring prefer­ences than for Laws+Ini­tial Con­di­tions though.

  If this is what you are say­ing, then I agree with the sec­ond and third points but dis­agree with the first—or at least, I don’t see any ar­gu­ment for it in A&M’s pa­per. It may still be true, but fur­ther ar­gu­ment is needed. In par­tic­u­lar their ar­gu­ments for (A) are pretty weak, me­thinks—that’s what my sec­tion “Ob­jec­tions to the ar­gu­ments for step 2” is about.

  Edit to clar­ify: By “I agree with the sec­ond point” I mean I agree that if the ar­gu­ment works at all, it prob­a­bly works for Laws+Ini­tial Con­di­tions as well. I don’t think the ar­gu­ment works though. But I do think that Oc­cam’s Ra­zor is prob­a­bly in­suffi­cient.

• Daniel Kokotajlo 7 Oct 2019 20:22 UTC

  3 points

  in reply to: Hazard's comment on: Hazard’s Short­form Feed

  Yep, agreed. I want all my friends and fam­ily to read the se­ries… and then have a con­ver­sa­tion with me about the ways in which it over­sim­plifies and mis­leads, in par­tic­u­lar the higher mind vs. prim­i­tive mind bit.

  On bal­ance though I think it’s great that it ex­ists and I pre­dict it will be the gate­way drug for a bunch of new ra­tio­nal­ists in years to come.

Oc­cam’s Ra­zor May Be Suffi­cient to In­fer the Prefer­ences of Ir­ra­tional Agents: A re­ply to Arm­strong & Mindermann

• Daniel Kokotajlo 7 Oct 2019 16:31 UTC

  1 point

  on: Ar­gu­ments for moral indefinability

  Would it be fair to say that moral in­defin­abil­ity is ba­si­cally what Yud­kowsky was talk­ing about with his slo­gan “Value is com­plex?”

  What about the stance of Par­tic­u­larism in moral philos­o­phy? On the face of it it seems very differ­ent, but I think it may be get­ting at a similar phe­nomenon.

• Daniel Kokotajlo 3 Oct 2019 13:11 UTC

  LW: 20 AF: 6

  AF

  on: Thoughts on Hu­man Models

  Wow, now I take the “But what if a bug puts a nega­tion on the util­ity func­tion” AGI failure mode more se­ri­ously:

  One of our code re­fac­tors in­tro­duced a bug which flipped the sign of the re­ward. Flip­ping the re­ward would usu­ally pro­duce in­co­her­ent text, but the same bug also flipped the sign of the KL penalty. The re­sult was a model which op­ti­mized for nega­tive sen­ti­ment while pre­serv­ing nat­u­ral lan­guage. Since our in­struc­tions told hu­mans to give very low rat­ings to con­tinu­a­tions with sex­u­ally ex­plicit text, the model quickly learned to out­put only con­tent of this form. This bug was re­mark­able since the re­sult was not gib­ber­ish but max­i­mally bad out­put. The au­thors were asleep dur­ing the train­ing pro­cess, so the prob­lem was no­ticed only once train­ing had finished. (From OpenAI https://​​ope­nai.com/​​blog/​​fine-tun­ing-gpt-2/​​)

  Might be worth adding a link to this epi­sode in the text?

• Daniel Kokotajlo 9 Sep 2019 14:04 UTC

  LW: 6 AF: 3

  AF

  on: Why Subagents?

  Thanks, this is re­ally cool!

  I’m a bit con­cerned about this sort of thing: “The sub­agents ar­gu­ment offers a the­o­ret­i­cal ba­sis for the idea that hu­mans have lots of in­ter­nal sub­agents, with com­pet­ing wants and needs, all con­stantly ne­go­ti­at­ing with each other to de­cide on ex­ter­nally-visi­ble be­hav­ior.”

  A worry I have about the stan­dard rep­re­sen­ta­tion the­o­rems is that they prove too much; if ev­ery­thing can be rep­re­sented as hav­ing a util­ity func­tion, then maybe it’s not so use­ful to talk about util­ity func­tions. Similarly now I worry: I thought when peo­ple talked about sub­agent the­o­ries of mind, they meant some­thing sub­stan­tial by this—not merely that the mind has in­com­plete (though still acyclic) prefer­ences!

• Daniel Kokotajlo 8 Sep 2019 16:23 UTC

  LW: 3 AF: 2

  AF

  in reply to: LukasM's comment on: Al­gorith­mic Similarity

  I’m glad you are in­ter­ested, and I’d love to hear your thoughts on the pa­per if you read it. I’d love to talk with you too; just send me an email when you’d like and we can skype or some­thing.

  What do you mean by “the more tech­ni­cal ver­sion of the prob­lem” ex­actly?

  My take right now is that al­gorith­mic similar­ity (and in­stan­ti­a­tion) at least the ver­sions of it rele­vant for con­scious­ness and de­ci­sion the­ory and episte­mol­ogy will have to be ei­ther a brute em­piri­cal fact about the world, or a sub­jec­tive fact about the mind of the agent rea­son­ing about it (like pri­ors and util­ity func­tions). What it will not be is some rea­son­ably non-ar­bi­trary prop­erty/​re­la­tion with in­ter­est­ing and use­ful prop­er­ties (like nash equil­ibria, cen­ters of mass, and tem­per­a­ture)

• Daniel Kokotajlo 27 Aug 2019 19:36 UTC

  LW: 7 AF: 4

  AF

  on: Al­gorith­mic Similarity

  Thanks, this is a good write-up!

  Many years ago I wrote my un­der­grad­u­ate the­sis on the wa­ter­fall prob­lem (though it went by an­other name to me). Ba­si­cally, I painstak­ingly and la­bo­ri­ously trans­formed an ar­bi­trary hu­man into an ar­bi­trary rock of suffi­cient size, via a se­ries of im­per­cep­ti­bly tiny steps none of which can be felt by the hu­man. (I did this in imag­i­na­tion, not in re­al­ity, to be clear) The point was to see if any of the steps seemed like good places to draw a line and say “Here, con­scious­ness is start­ing to go out; the sys­tem is start­ing to be less of a per­son.” As a re­sult I be­came fairly con­vinced that there aren’t any good places to draw the line. So I guess I’m a wa­ter­fall apol­o­gist now!

• Daniel Kokotajlo 27 Aug 2019 3:19 UTC

  LW: 13 AF: 6

  AF

  on: Six AI Risk/​Strat­egy Ideas

  I par­tic­u­larly like your “Log­i­cal vs. phys­i­cal risk aver­sion” dis­tinc­tion, and agree that we should pri­ori­tize re­duc­ing log­i­cal risk. I think acausal trade makes this par­tic­u­larly con­crete. If we make a mis­al­igned su­per­in­tel­li­gence that “plays nice” in the acausal bar­gain­ing com­mu­nity I’d think that’s bet­ter than mak­ing an al­igned su­per­in­tel­li­gence that doesn’t, be­cause over­all it mat­ters far more that the com­mu­nity is nice than that it have a high pop­u­la­tion of peo­ple with our val­ues.

  I also re­ally like your point about how pro­vid­ing ev­i­dence that AI safety is difficult may be one of the most im­por­tant rea­sons to do AI safety re­search. I guess I’d like to see some em­piri­cally grounded anal­y­sis of how likely it is that the rele­vant poli­cy­mak­ers and so forth will be swayed by such things. So far it seems like they’ve been swayed by di­rect ar­gu­ments that the prob­lem is hard, and not so much by our failures to make progress. If any­thing failure of AI safety re­searchers to make progress seems to en­courage their crit­ics.