Daniel Kokotajlo

• Daniel Kokotajlo 3 Apr 2020 20:57 UTC

  5 points

  in reply to: maximkazhenkov's comment on: Im­pli­ca­tions of the Dooms­day Ar­gu­ment for x-risk reduction

  Agreed. If you want to talk more about these ideas some­time, I’d be happy to video chat!

• Daniel Kokotajlo 3 Apr 2020 15:32 UTC

  5 points

  in reply to: maximkazhenkov's comment on: Im­pli­ca­tions of the Dooms­day Ar­gu­ment for x-risk reduction

  re: in­verse pro­porti­nal­ity: Good point, I’ll have to think about that more. Maybe it does neatly can­cel out, or even worse, since my util­ity func­tion isn’t lin­ear in happy lives lived, maybe it more than can­cels out.

  I for one have se­ri­ously in­ves­ti­gated all those weird philo­soph­i­cal ideas you men­tioned. ;) And I think our com­mu­nity has been pretty good about tak­ing these ideas se­ri­ously, es­pe­cially com­pared to, well, liter­ally ev­ery other com­mu­nity, in­clud­ing aca­demic philos­o­phy. Our over­ton win­dow definitely in­cludes all these ideas, I’d say.

  But I agree with your gen­eral point that there is a ten­sion we should ex­plore. Even if we are OK se­ri­ously dis­cussing these ideas, we of­ten don’t ac­tu­ally live by them. Our over­ton win­dow in­cludes them, but our me­dian opinion doesn’t. Why not?

  I think there is a good an­swer, and it has to do with hu­mil­ity/​cau­tion. Philos­o­phy is weird. If you fol­low ev­ery ar­gu­ment where it leads you, you very quickly find that your be­liefs don’t add up to nor­mal­ity, or any­thing close. Faith that be­liefs will (ap­prox­i­mately) add up to nor­mal­ity seems to be im­por­tant for stay­ing sane and pro­duc­tive, and more­over, seems to have been vin­di­cated of­ten in the past: crazy-sound­ing ar­gu­ments turn out to have flaws in them, or maybe they work but there is an ad­di­tional ar­gu­ment we hadn’t con­sid­ered that com­bines with it to add up to nor­mal­ity.

• Daniel Kokotajlo 3 Apr 2020 12:38 UTC

  3 points

  in reply to: Lanrian's comment on: Im­pli­ca­tions of the Dooms­day Ar­gu­ment for x-risk reduction

  Yeah. It de­pends on how you define ex­tinc­tion. I agree that most simu­la­tions don’t last very long. (You don’t even need the dooms­day ar­gu­ment to get that con­clu­sion, I think)

• Daniel Kokotajlo 3 Apr 2020 1:39 UTC

  4 points

  on: Im­pli­ca­tions of the Dooms­day Ar­gu­ment for x-risk reduction

  I’d like to see some­one ex­plore the ap­par­ent con­tra­dic­tion in more de­tail. Even if I were con­vinced that we will al­most cer­tainly fail, I might still pri­ori­tize x-risk re­duc­tion, since the stakes are so high.

  Any­how, my guess is that most peo­ple think the dooms­day ar­gu­ment prob­a­bly doesn’t work. I am not sure my­self. If it does work though, its con­clu­sion is not that we will all go ex­tinct soon, but rather that an­ces­tor simu­la­tions are one of the main uses of cos­mic re­sources.

• Daniel Kokotajlo 1 Apr 2020 14:29 UTC

  9 points

  on: What achieve­ments have peo­ple claimed will be warn­ing signs for AGI?

  AI Im­pacts has a list of rea­sons peo­ple give for why cur­rent meth­ods won’t lead to hu­man-level AI. With sources. It’s not ex­actly what you are look­ing for, but it’s close, be­cause most of these could be in­verted and used as warn­ing signs for AGI, e.g. “Cur­rent meth­ods can’t build good, ex­plana­tory causal mod­els” be­comes “When we have AI which can build good, ex­plana­tory causal mod­els, that’s a warn­ing sign.”

• Daniel Kokotajlo 1 Apr 2020 13:28 UTC

  4 points

  on: Call for vol­un­teers: as­sess­ing Kurzweil, 2019

  I’d be happy to vol­un­teer a bit. I don’t have much time, but this sounds fun, so maybe I could do a few.

• Daniel Kokotajlo 1 Apr 2020 10:50 UTC

  4 points

  in reply to: Tetraspace Grouping's comment on: Blog Post Day II Retrospective

  OK, so you + Gy­ro­diot are mak­ing me think maybe I should do an­other one soon. But to be hon­est I need to fo­cus less on blog­ging and more on work­ing for a bit, so I per­son­ally won’t be ready for at least a few weeks I think.

  When­ever it hap­pens, I should sched­ule it far in ad­vance I think. That way peo­ple have more of a chance to find out about it.

• Daniel Kokotajlo 31 Mar 2020 20:01 UTC

  LW: 2 AF: 1

  AF

  in reply to: evhub's comment on: Three Kinds of Competitiveness

  Oh right, how could I for­get! This makes me very happy. :D

• Daniel Kokotajlo 31 Mar 2020 17:42 UTC

  LW: 3 AF: 2

  AF

  in reply to: ofer's comment on: Three Kinds of Competitiveness

  Good point about in­ner al­ign­ment prob­lems be­ing a blocker to date-com­pet­i­tive­ness for IDA… but aren’t they also a blocker to date-com­pet­i­tive­ness for ev­ery other al­ign­ment scheme too pretty much? What al­ign­ment schemes don’t suffer from this prob­lem?

  I’m think­ing “Do any­thing use­ful that a hu­man with a lot of time can do” is go­ing to be sub­stan­tially less ca­pa­ble than full-blown su­per­in­tel­li­gent AGI. How­ever, that’s OK, be­cause we can use IDA as a step­ping-stone to that. IDA gets us an al­igned sys­tem sub­stan­tially more ca­pa­ble than a hu­man, and we use that sys­tem to solve the al­ign­ment prob­lem and build some­thing even bet­ter.

  It’s in­ter­est­ing how Paul ad­vo­cates merg­ing cost and perfor­mance-com­pet­i­tive­ness, and you ad­vo­cate merg­ing perfor­mance and date-com­pet­i­tive­ness. I think it’s fine to just talk about “com­pet­i­tive­ness” full stop, and only bother to spec­ify what we mean more pre­cisely when needed. Some­times we’ll mean one of the three, some­times two of the three, some­times all three.

• Daniel Kokotajlo 31 Mar 2020 17:34 UTC

  2 points

  in reply to: Pattern's comment on: Three Kinds of Competitiveness

  Yes. An up­grade to an AI that makes it run faster, with no side-effects, would be an im­prove­ment to both perfor­mance and cost-com­pet­i­tive­ness.

• Daniel Kokotajlo 31 Mar 2020 17:33 UTC

  LW: 2 AF: 1

  AF

  in reply to: paulfchristiano's comment on: Three Kinds of Competitiveness

  I knew that the goal was to get IDA to be cost-com­pet­i­tive, but I thought cur­rent ver­sions of it weren’t. But that was just my rough im­pres­sion; glad to be wrong, since it makes IDA seem even more promis­ing. :) Of all the pro­pos­als I’ve heard of, IDA seems to have the best com­bi­na­tion of cost, date, and perfor­mance-com­pet­i­tive­ness.

• Daniel Kokotajlo 31 Mar 2020 17:29 UTC

  LW: 2 AF: 1

  AF

  in reply to: paulfchristiano's comment on: Three Kinds of Competitiveness

  I agree this may be true in most cases, but the chance of it not be­ing true for AI is large enough to mo­ti­vate the dis­tinc­tion. Be­sides, not all cases in which perfor­mance and cost can be traded off are the same; in some sce­nar­ios the “price” of perfor­mance is very high whereas in other sce­nar­ios it is low. (e.g. in Grad­ual Eco­nomic Takeover, let’s say, a sys­tem be­ing twice as qual­i­ta­tively in­tel­li­gent could be equiv­a­lent to be­ing a quar­ter the price. Whereas in Fi­nal Con­flict, a sys­tem twice as qual­i­ta­tively in­tel­li­gent would be equiv­a­lent to be­ing one per­cent the price.) So if we are think­ing of a sys­tem as “com­pet­i­tive with X% over­head,” well, X% is go­ing to vary tremen­dously de­pend­ing on which sce­nario is re­al­ized. Seems worth say­ing e.g. “costs Y% more com­pute, but is Z% more ca­pa­ble.”

Blog Post Day II Retrospective

• Daniel Kokotajlo 31 Mar 2020 11:34 UTC

  LW: 2 AF: 1

  AF

  in reply to: evhub's comment on: Three Kinds of Competitiveness

  Mmm, nice. Thanks! I like your dis­tinc­tion also. I think yours is suffi­ciently differ­ent that we shouldn’t see the two sets of dis­tinc­tions as com­pet­ing.* A sys­tem which has an ob­jec­tive which would be ca­pa­ble on pa­per but isn’t ca­pa­ble in prac­tice due to in­ner-al­ign­ment failures would be perfor­mance-un­com­pet­i­tive but ob­jec­tive-com­pet­i­tive. For this rea­son I think we shouldn’t equate ob­jec­tive and perfor­mance com­pet­i­tive­ness.

  If op­er­at­ing an AI sys­tem turns out to be an im­por­tant part of the cost, then cost+date com­pet­i­tive­ness would turn out to be differ­ent from train­ing com­pet­i­tive­ness, be­cause cost com­pet­i­tive­ness in­cludes what­ever the rele­vant costs are. How­ever I ex­pect op­er­at­ing costs will be much less rele­vant to con­trol­ling the fu­ture than costs in­curred dur­ing the cre­ation of the sys­tem (all that train­ing, data-gath­er­ing, in­fras­truc­ture build­ing, etc.) so I think the map­ping be­tween cost+date com­pet­i­tive­ness and train­ing com­pet­i­tive­ness ba­si­cally works.

  *In­so­far as they are com­pet­ing, I still pre­fer mine; as you say, it ap­plies to more than just pro­saic AI al­ign­ment pro­pos­als. More­over, it makes it eas­ier for us to talk about com­pe­ti­tions as well, e.g. “In the FOOM sce­nario we need to win a date com­pe­ti­tion; cost-com­pet­i­tive­ness still mat­ters but not as much.” More­over cost, perfor­mance, and date are fairly self-ex­plana­tory terms, whereas as you point out “ob­jec­tive” is more opaque. More­over I think it’s worth dis­t­in­guish­ing be­tween cost and date com­pet­i­tive­ness; in some sce­nar­ios one will be much more im­por­tant than the other, and of course the two kinds of com­pet­i­tive­ness vary in­de­pen­dently in AI safety schemes (in­deed maybe they are mildly anti-cor­re­lated? Some schemes are fairly well-defined and cod­ified already, but would re­quire tons of com­pute, whereas other schemes are more vague and thus would re­quire tons of tweak­ing and cau­tious test­ing to get right, but don’t take that much com­pute. I do like how your ver­sion maps more onto the in­ner vs. outer al­ign­ment dis­tinc­tion.

• Daniel Kokotajlo 31 Mar 2020 1:14 UTC

  LW: 2 AF: 1

  AF

  on: Three Kinds of Competitiveness

  Some thoughts that came to me af­ter I wrote this post:

  --I’m not sure I should define date-com­pet­i­tive the way I do. Maybe in­stead of “can be built” it should be “is built.” If we go the lat­ter route, the FOOM sce­nario is an ex­tremely in­tense date com­pe­ti­tion. If we go the former route, the FOOM sce­nario is not nec­es­sar­ily an in­tense date com­pe­ti­tion; it de­pends on what other fac­tors are at play. For ex­am­ple, maybe there are only a few ma­jor AI pro­jects and all of them are pretty so­cially re­spon­si­ble, so a de­sign is more likely to win if it can be built sooner, but it won’t nec­es­sar­ily win; maybe cooler heads will pre­vail and build a safer de­sign in­stead.

  --Why is date-com­pet­i­tive­ness worth call­ing a kind of com­pet­i­tive­ness at all? Why not just say: “We want our AI safety scheme/​de­sign to be cost- and perfor­mance-com­pet­i­tive, and also we need to be able to build it fairly quickly com­pared to the other stuff that gets built.” Well, 1. Even that is clunky and awk­ward com­pared to the el­e­gant ”...and also date-com­pet­i­tive.” 2. It re­ally does have the com­par­a­tive fla­vor of com­pe­ti­tion to it; what mat­ters is not how long it takes us to com­plete our safety scheme, but how long it takes rel­a­tive to un­al­igned schemes, and it’s not as sim­ple as just “we need to be first,” rather it’s that sooner is bet­ter but do­ing it later isn’t nec­es­sar­ily game over… 3. It seems to be use­ful for de­scribing date com­pe­ti­tions, which are im­por­tant to dis­t­in­guish from situ­a­tions which are not date com­pe­ti­tions or less so. (Aside: A clas­sic crit­i­cism of the “Let’s build up­loads first, and up­load peo­ple we trust” strat­egy is that neu­ro­mor­phic AI will prob­a­bly come be­fore up­loads. In other words, this strat­egy is not date-com­pet­i­tive.)

  --I’m toy­ing with the idea of adding “al­ign­ment-com­pet­i­tive­ness” (mean­ing, as al­igned or more al­igned than com­pet­ing sys­tems) and “al­ign­ment com­pe­ti­tion” to the set of defi­ni­tions. This sounds silly, but it would be con­cep­tu­ally neat, be­cause then we can say: We hope for sce­nar­ios in which con­trol of the fu­ture is a very in­tense al­ign­ment com­pe­ti­tion, and we are work­ing hard to make it that way. ”

Three Kinds of Competitiveness

• Daniel Kokotajlo 29 Mar 2020 11:33 UTC

  LW: 8 AF: 3

  AF

  on: Any re­but­tals of Chris­ti­ano and AI Im­pacts on take­off speeds?

  Just wanna say, I in­tend to get around to writ­ing re­but­tals some­day. I definitely have sev­eral coun­ter­ar­gu­ments in mind; the force­ful take­downs you men­tion weren’t very con­vinc­ing to me, though they did make me up­date away from fast take­off.

• Daniel Kokotajlo 28 Mar 2020 23:30 UTC

  LW: 2 AF: 1

  AF

  in reply to: Ben Pace's comment on: Ben Pace’s Short­form Feed

  Well, that wasn’t the sce­nario I had in mind. The sce­nario I had in mind was: Peo­ple in the year 2030 pass a law re­quiring fu­ture gov­ern­ments to make an­ces­tor simu­la­tions with happy af­ter­lives, be­cause that way it’s prob­a­ble that they them­selves will be in such a simu­la­tion. (It’s like cry­on­ics, but cheaper!) Then, hun­dreds or billions of years later, the fu­ture gov­ern­ment car­ries out the plan, as re­quired by law.

  Not say­ing this is what we should do, just say­ing it’s a de­ci­sion I could sym­pa­thize with, and I imag­ine it’s a de­ci­sion some frac­tion of peo­ple would make, if they thought it was an op­tion.

• Daniel Kokotajlo 28 Mar 2020 17:48 UTC

  5 points

  in reply to: rohinmshah's comment on: When to as­sume neu­ral net­works can solve a problem

  In­ter­est­ing. Well, I imag­ine you don’t have the time right now, but I just want to reg­ister that I’d love to hear more about this. What ques­tion­able as­sump­tions does Su­per­in­tel­li­gence make, that aren’t made by Hu­man Com­pat­i­ble? (This re­quest for info goes out to ev­ery­one, not just Ro­hin)

Re­minder: Blog Post Day II to­day!