Daniel Kokotajlo

• Daniel Kokotajlo 21 Feb 2020 15:47 UTC

  LW: 2 AF: 2

  AF

  in reply to: steve2152's comment on: On un­fix­ably un­safe AGI architectures

  I agree that AGI is more omni-use than bioweapons and thus will be harder to get peo­ple not to de­velop and use. I think our prospects look pretty bleak in this sce­nario, but it’s not com­pletely hope­less.

  For hu­man clon­ing, what I had in mind was a na­tion clon­ing its smartest in­di­vi­d­u­als for the pur­pose of hav­ing bet­ter sci­ence/​tech. Think of what the US could have ac­com­plished if they had 10,000 Von Neu­manns in­stead of 1.

• Daniel Kokotajlo 20 Feb 2020 19:14 UTC

  LW: 3 AF: 2

  AF

  on: On un­fix­ably un­safe AGI architectures

  Even as a se­ries of worse and worse AGI ac­ci­dents oc­cur, wih out-of-con­trol AGIs self-repli­cat­ing around the in­ter­net etc., a few peo­ple will keep try­ing to fix the un­fix­able AGI, see­ing this as the only path to get this slow-rol­ling catas­tro­phe un­der con­trol (while ac­tu­ally mak­ing it worse).

  Maybe at this point there would be the poli­ti­cal will for a But­le­rian Jihad. ;) Or more se­ri­ously, a self-im­posed ban on AGI similar to the cur­rent self-im­posed bans on hu­man clon­ing and biolog­i­cal weapons. I agree this is a long shot given our cur­rent ex­pe­rience with cli­mate change, but still, it seems pos­si­ble. And per­haps the AGI ac­ci­dents would be more news­wor­thy and grip­ping than cli­mate change is, mak­ing it eas­ier to rouse the pub­lic.

• Daniel Kokotajlo 20 Feb 2020 18:59 UTC

  LW: 3 AF: 1

  AF

  on: Tes­sel­lat­ing Hills: a toy model for demons in im­perfect search

  This is awe­some, thanks!

  So, to check my un­der­stand­ing: You have set up a sort of ar­tifi­cial feed­back loop, where there are N over­lap­ping pat­terns of hills, and each one gets stronger the farther you travel in a par­tic­u­lar di­men­sion/​di­rec­tion. So if one or more of these pat­terns tends sys­tem­at­i­cally to push the ball in the same di­rec­tion that makes it stronger, you’ll get a feed­back loop. And then there is se­lec­tion be­tween pat­terns, in the sense that the pat­tern which pushes the strongest will beat the ones that push more weakly, even if both have feed­back loops go­ing.

  And then the ar­gu­ment is, even though these feed­back loops were ar­tifi­cial /​ baked in by you, in “nat­u­ral” search prob­lems there might be a similar situ­a­tion… what ex­actly is the rea­son for this? I guess my con­fu­sion is in whether to ex­pect real life prob­lems to have this prop­erty where mov­ing in a par­tic­u­lar di­rec­tion strength­ens a par­tic­u­lar pat­tern. One way I could see this hap­pen­ing is if the pat­terns are them­selves pretty smart, and are able to sense which di­rec­tions strengthen them at any given mo­ment. Or it could hap­pen if, by chance, there hap­pens to be a di­rec­tion and a pat­tern such that the pat­tern sys­tem­at­i­cally pushes in that di­rec­tion and the di­rec­tion sys­tem­at­i­cally strength­ens that pat­tern. But how likely are these? I don’t know. I guess your case is a case of the sec­ond, but it’s rigged a bit, be­cause of how you built in the sys­tem­atic-strength­en­ing effect.

  Am I fol­low­ing, or am I mi­s­un­der­stand­ing?

• Daniel Kokotajlo 19 Feb 2020 16:08 UTC

  26 points

  on: Is there an in­tu­itive way to ex­plain how much bet­ter su­perfore­cast­ers are than reg­u­lar fore­cast­ers?

  Like ig­no­ran­ceprior said, my AI Im­pacts post has three in­tu­itive ways of think­ing about the re­sults:

  
  

  Way One: Let’s calcu­late some ex­am­ples of pre­dic­tion pat­terns that would give you Brier scores like those men­tioned above. Sup­pose you make a bunch of pre­dic­tions with 80% con­fi­dence and you are cor­rect 80% of the time. Then your Brier score would be 0.32, roughly mid­dle of the pack in this tour­na­ment. If in­stead it was 93% con­fi­dence cor­rect 93% of the time, your Brier score would be 0.132, very close to the best su­perfore­cast­ers and to GJP’s ag­gre­gated fore­casts.14 In these ex­am­ples, you are perfectly cal­ibrated, which helps your score—more re­al­is­ti­cally you would be im­perfectly cal­ibrated and thus would need to be right even more of­ten to get those scores.

  Way Two: “An al­ter­na­tive mea­sure of fore­cast ac­cu­racy is the pro­por­tion of days on which fore­cast­ers’ es­ti­mates were on the cor­rect side of 50%. … For all ques­tions in the sam­ple, a chance score was 47%. The mean pro­por­tion of days with cor­rect es­ti­mates was 75%…”15 Ac­cord­ing to this chart, the su­perfore­cast­ers were on the right side of 50% al­most all the time:16

  Way Three: “Across all four years of the tour­na­ment, su­perfore­cast­ers look­ing out three hun­dred days were more ac­cu­rate than reg­u­lar fore­cast­ers look­ing out one hun­dred days.”17 (Bear in mind, this wouldn’t nec­es­sar­ily hold for a differ­ent genre of ques­tions. For ex­am­ple, in­for­ma­tion about the weather de­cays in days, while in­for­ma­tion about the cli­mate lasts for decades or more.)

• Daniel Kokotajlo 19 Feb 2020 11:33 UTC

  1 point

  on: Ex­cite­ment vs childishness

  I’m glad you made that dis­claimer. In our com­mu­nity—LW—the ra­tio of ex­cite­ment/​childish­ness bias is prob­a­bly un­usu­ally high, per­haps even high enough that we need to be on higher guard against ex­cite­ment. But in the wider com­mu­nity of “smart and/​or im­por­tant peo­ple think­ing about AI” it seems pretty clear that childish­ness bias is much much stronger.

• Daniel Kokotajlo 19 Feb 2020 11:29 UTC

  1 point

  in reply to: Daniel Kokotajlo's comment on: Ex­cite­ment vs childishness

  Like, I agree that Michael Wulf­son is a per­son for whom the ex­cite­ment bias was stronger, judg­ing by his story. But even for him, the childish­ness bias was su­per strong too, and even­tu­ally won out. Most peo­ple are not like him; most peo­ple will find the childish­ness bias /​ ex­cite­ment bias ra­tio much higher than he did.

• Daniel Kokotajlo 19 Feb 2020 11:23 UTC

  1 point

  in reply to: Daniel Kokotajlo's comment on: Ex­cite­ment vs childishness

  In­deed I’d go so far as to say if this is the best case we can think of of ex­cite­ment bias be­ing stronger, that’s pretty good ev­i­dence that in fact childish­ness bias is usu­ally way stronger.

• Daniel Kokotajlo 19 Feb 2020 11:22 UTC

  6 points

  in reply to: Matthew Barnett's comment on: Ex­cite­ment vs childishness

  I’m not sure this is strong coun­terev­i­dence, be­cause it can be in­ter­preted as a case of childish­ness bias over­com­ing ex­cite­ment bias:

  But there were also coun­ter­vailing effects in my mind, lead­ing away from the god sce­nario. The strongest was the out­landish­ness of it all. I had always been dis­mis­sive of ideas that seem like dooms­day the­o­ries, so I wasn’t au­to­mat­i­cally com­fortable giv­ing the god sce­nario cre­dence in my mind. I was hes­i­tant to in­tro­duce the idea to peo­ple who I thought might draw nega­tive con­clu­sions about my judge­ment.

• Daniel Kokotajlo 19 Feb 2020 1:54 UTC

  LW: 1 AF: 1

  AF

  in reply to: KatjaGrace's comment on: Soft take­off can still lead to de­ci­sive strate­gic advantage

  I like your point #2; I should think more about how the 30 year num­ber changes with size. Ob­vi­ously it’s smaller for big­ger en­tities and big­ger for smaller en­tities, but how much? E.g. if we tele­ported 2020 Es­to­nia back into 1920, would it be able to take over the world? Prob­a­bly. What about 1970 though? Less clear.

  Mili­tary power isn’t what I’m get­ting at ei­ther, at least not if mea­sured in the way that would re­sult in AI com­pa­nies hav­ing lit­tle of it. Cortez had, maybe, ¹⁄₁₀,000th of the mil­i­tary power of Mex­ico when he got started. At least if you mea­sure in ways like “What would hap­pen if X fought Y.” Prob­a­bly ¹⁄₁₀,000th of Mex­ico’s mil­i­tary could have defeated Cortez’ ini­tial band.

  If we try to model Cortez’ takeover as him hav­ing more of some met­ric than all of Mex­ico had, then pre­sum­ably Spain had sev­eral or­ders of mag­ni­tude more of that met­ric than Cortez did, and Western Europe as a whole had at least an or­der of mag­ni­tude more than that. So Western Europe had *many* or­ders of mag­ni­tude more of this stuff, what­ever it is, than Mex­ico, even though Mex­ico had a similar pop­u­la­tion and GDP. So they must have been grow­ing much faster than Mex­ico for quite some time to build up such a lead—and this was be­fore the in­dus­trial rev­olu­tion! More gen­er­ally, this met­ric that is used for pre­dict­ing takeovers seems to be the sort of thing that can grow and/​or shrink or­ders of mag­ni­tude very quickly, as illus­trated by the var­i­ous cases through­out his­tory of small groups from back­wa­ter re­gions tak­ing over rich em­pires.

  (Warn­ing: I’m pul­ling these claims out of my ass, I’m not a his­to­rian, I might be to­tally wrong. I should look up these num­bers.)

• Daniel Kokotajlo 18 Feb 2020 19:37 UTC

  3 points

  in reply to: jacobjacob's comment on: Blog Post Day (Unoffi­cial)

  Glad to hear you are in­ter­ested! Well, I’m in US Eastern time, but timing can be flex­ible. If we have enough peo­ple, per­haps Blog Post Day will effec­tively be longer than 24 hours. I’m think­ing it would be a rel­a­tively ca­sual af­fair, with peo­ple drop­ping in or out as they see fit.

• Daniel Kokotajlo 18 Feb 2020 19:07 UTC

  3 points

  on: Blog Post Day (Unoffi­cial)

  Thanks in ad­vance to those who join me on this ven­ture! And those who give ad­vice, crit­i­cism, etc.

  If we get suffi­cient in­ter­est, it might be good to or­ga­nize lo­cal mee­tups. Any­one else in North Carolina want to meet up with me for this?

Blog Post Day (Unoffi­cial)

• Daniel Kokotajlo 18 Feb 2020 16:27 UTC

  LW: 1 AF: 1

  AF

  in reply to: KatjaGrace's comment on: Soft take­off can still lead to de­ci­sive strate­gic advantage

  I was think­ing of an ini­tially large coun­try grow­ing fast via AI, yes. Still counts; it is soft take­off lead­ing to DSA. How­ever I am also mak­ing much stronger claims than that—I think it could hap­pen with a cor­po­ra­tion or rogue AGI.

  I don’t think an­nual in­come is at all a good mea­sure of how close an en­tity is to tak­ing over the world. When Cortez landed in Mex­ico he had less than ¹⁄₁₀₀,000th of the in­come, pop­u­la­tion, etc. of the re­gion, yet he ruled the whole place three years later. Then a few years af­ter that Pizarro re­peated the feat in Peru, good ev­i­dence that it wasn’t just an amaz­ing streak of luck.

• Daniel Kokotajlo 16 Feb 2020 14:42 UTC

  9 points

  on: A ‘Prac­tice of Ra­tion­al­ity’ Se­quence?

  I nom­i­nate this thing john­swent­worth did. In ad­di­tion to the rea­sons he gives, I’ll add that be­ing able to learn on your own, quickly, seems like a good skill to have, and re­lated to (though maybe not the same thing as) ra­tio­nal­ity.

• Daniel Kokotajlo 16 Feb 2020 14:32 UTC

  3 points

  on: Why Science is slow­ing down, Univer­si­ties and Maslow’s hi­er­ar­chy of needs

  I think I find your over­all con­clu­sion plau­si­ble, but I think your ar­gu­ment for it in places was du­bi­ous:

  But again, even if you as­sume I’m wrong, that still leave us with uni­ver­si­ties that strug­gle to op­ti­mize for 2, 3 and maybe 4, los­ing out on 5 in the pro­cess.

  One could in­stead in­ter­pret the situ­a­tion as: Univer­si­ties are op­ti­miz­ing hard for 5, and as a re­sult they are un­der­stand­ably los­ing out on 2, 3, and 4 in the pro­cess.

  In­deed, I think there is some­thing to be said for this. A few years ago I half-jok­ingly wrote a pa­per ti­tled “Kal­lipo­lis, USA,” in which I ar­gue that the pre­sent-day USA is in fact Plato’s ideal state.

  A big part of my ar­gu­ment was the way in which the uni­ver­sity sys­tem works. In par­tic­u­lar, (in con­junc­tion with the rest of so­ciety) it seems to be op­ti­miz­ing pretty hard to get peo­ple to “fol­low their pas­sion,” and in par­tic­u­lar by forc­ing ev­ery­one to go to col­lege and take gen-ed re­quire­ments ar­guably the sys­tem is do­ing the best it can to scout and re­cruit peo­ple who are suited to the priest­hood/​academia.

• Daniel Kokotajlo 15 Feb 2020 12:19 UTC

  LW: 3 AF: 2

  AF

  on: Bayesian Evolv­ing-to-Extinction

  How nec­es­sary is it that there be an ex­plicit side-chan­nel? Could you not get the same re­sults in the stan­dard situ­a­tion in which an agent is se­lect­ing ac­tions on the ba­sis of ex­pected util­ity?

• Daniel Kokotajlo 12 Feb 2020 19:49 UTC

  LW: 6 AF: 3

  AF

  in reply to: johnswentworth's comment on: De­mons in Im­perfect Search

  Ah, that does help, thanks. In my words: A search pro­cess that is vuln­er­a­ble to lo­cal min­ima doesn’t nec­es­sar­ily con­tain a sec­ondary search pro­cess, be­cause it might not be sys­tem­at­i­cally com­par­ing lo­cal min­ima and choos­ing be­tween them ac­cord­ing to some crite­ria. It just goes for the first one it falls for, or maybe slightly more nu­anced, the first suffi­ciently big one it falls for.

  By con­trast, in the ball rol­ling ex­am­ple you gave, the walls/​ridges were com­pet­ing with each other, such that the “best” one (or some­thing like that) would be sys­tem­at­i­cally se­lected by the ball, rather than just the first one or the first-suffi­ciently-big one.

  So in that case, look­ing over your list again...

  OK, I think I see how or­ganic life aris­ing from chem­istry is an ex­am­ple of a sec­ondary search pro­cess. It’s not just a lo­cal min­ima that chem­istry found it­self in, it’s a big com­pe­ti­tion be­tween differ­ent kinds of lo­cal min­ima. And now I think I see how this would go in the other ex­am­ples too. As I origi­nally said in my top-level com­ment, I’m not sure this ap­plies to the ex­am­ple I brought up, ac­tu­ally. Would the “Insert my name as the au­thor of all use­ful heuris­tics” heuris­tic be out­com­peted by some­thing else even­tu­ally, or not? I bet not, which in­di­cates that it’s a “mere” lo­cal min­ima and not one that is part of a broader sec­ondary search pro­cess.

• Daniel Kokotajlo 12 Feb 2020 18:35 UTC

  3 points

  in reply to: Daniel Kokotajlo's comment on: Si­mu­la­tion of tech­nolog­i­cal progress (work in progress)

  I should add though that I haven’t sys­tem­at­i­cally ex­am­ined these graphs yet, so it’s pos­si­ble I’m just miss­ing some­thing—e.g. it oc­curs to me right now that maybe some of these graphs I saw were re­ally lo­gis­tic func­tions rather than hy­per­bolic or ex­po­nen­tial-un­til-you-hit-limits. I should make some more and look at them more care­fully.

• Daniel Kokotajlo 12 Feb 2020 18:30 UTC

  7 points

  in reply to: SDM's comment on: Si­mu­la­tion of tech­nolog­i­cal progress (work in progress)

  Yes, thanks! I mostly agree with that as­sess­ment,* though as an aside I have a beef with the im­pli­ca­tion that Bostrom, Yud­kowsky, etc. ex­pect dis­con­ti­nu­ities. That beef is with Paul Chris­ti­ano, not you. :)

  So far the biggest up­date this has been for me, I think, is that it seems to have shown that it’s quite pos­si­ble to get an in­tel­li­gence ex­plo­sion even with­out eco­nomic feed­back loops. Like, even with a fixed com­pute/​money bud­get—or even with a fixed num­ber of sci­en­tists and fixed amount of re­search fund­ing—we could get sin­gu­lar­ity. At least in prin­ci­ple. This is weird be­cause in prac­tice I am pretty sure I re­mem­ber read­ing that the growth we’ve seen so far can be best ex­plained via an eco­nomic feed­back loop: Bet­ter tech­nol­ogy al­lows for big­ger pop­u­la­tion and econ­omy which al­lows for more sci­en­tists and fund­ing which al­lows for bet­ter tech­nol­ogy. So I’m a bit con­fused, I must say—my model is giv­ing me re­sults I would have pre­dicted wouldn’t hap­pen.

  *There have been a few cases where the growth didn’t look hy­per­bolic, but rather like a steady ex­po­nen­tial trend that then turns into a sin­gu­lar­ity. World GDP, by con­trast, has what looks like at least three ex­po­nen­tial trends in it, such that it is more par­si­mo­nious to model it as hy­per­bolic growth. I think.

• Daniel Kokotajlo 12 Feb 2020 11:47 UTC

  LW: 5 AF: 3

  AF

  in reply to: johnswentworth's comment on: De­mons in Im­perfect Search

  Hmmm, this doesn’t work to dis­t­in­guish the two for me. Couldn’t you say a lo­cal min­ima in­volves a sec­ondary op­ti­miz­ing search pro­cess that has that min­ima as its ob­jec­tive? To use your ball anal­ogy, what ex­actly is the differ­ence be­tween these twisty de­mon hills and a sim­ple crater-shaped pit? (Or, what is the differ­ence be­tween a search pro­cess that is vuln­er­a­ble to twisty de­mon hills and one which is vuln­er­a­ble to pits?)