Daniel Kokotajlo

Philosophy PhD student, worked at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Not sure what I’ll do next yet. Views are my own & do not represent those of my current or former employer(s). I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

What 2026 looks like

Ta­boo “Out­side View”

Deep­Mind: Gen­er­ally ca­pa­ble agents emerge from open-ended play

Fun with +12 OOMs of Compute

Birds, Brains, Planes, and AI: Against Ap­peals to the Com­plex­ity/​Mys­te­ri­ous­ness/​Effi­ciency of the Brain

Cortés, Pizarro, and Afonso as Prece­dents for Takeover

Deep­mind’s Gato: Gen­er­al­ist Agent

The Com­mit­ment Races problem

The date of AI Takeover is not the day the AI takes over

Against GDP as a met­ric for timelines and take­off speeds

Soft take­off can still lead to de­ci­sive strate­gic advantage

[Question] What will 2040 prob­a­bly look like as­sum­ing no sin­gu­lar­ity?

The Main Sources of AI Risk?

• Daniel Kokotajlo 8 Oct 2022 10:15 UTC

  99 points

  83

  on: Why I think there’s a one-in-six chance of an im­mi­nent global nu­clear war

  Last spring, I predicted that once loss of occupied territory loomed, he would annex what he controlled and start talking about nuclear defense of Russia’s new borders – and here we are.

  Citation/​link please? “Trust, but verify.”

  I view it as highly unlikely (<10%) that Putin would accept “Vietnam” without first going nuclear, because it would almost certainly result in him being overthrown and jailed or killed.

  This seems overconfident to me. So many people made confident predictions about what Putin would or wouldn’t do (or Trump, or Obama, or Xi for that matter) and were wrong. It’s hard to predict world leader behavior even in cases with somewhat of a precedent, which this is not. Example: Early in this war I thought “Belarus will almost certainly join the war sooner or later, because Putin can have Lukashenko assassinated, along with loads of higher-ups in the government, if he wants.” Welp, I was wrong. Don’t know why yet. Similarly I think that if Putin were to accept a “vietnam” he’d probably still remain in power. You might think that there’d be a revolution, and there totally might, but I don’t know how you can be almost certain.
  
  

DALL-E by OpenAI

Linkpost: Github Copi­lot pro­duc­tivity experiment

Im­manuel Kant and the De­ci­sion The­ory App Store

Re­view of Soft Take­off Can Still Lead to DSA

• Daniel Kokotajlo 21 Feb 2024 19:55 UTC

  76 points

  18

  on: Daniel Koko­ta­jlo’s Shortform

  1. Probably there will be AGI soon—literally any year now.

  2. Probably whoever controls AGI will be able to use it to get to ASI shortly thereafter—maybe in another year, give or take a year.

  3. Probably whoever controls ASI will have access to a spread of powerful skills/​abilities and will be able to build and wield technologies that seem like magic to us, just as modern tech would seem like magic to medievals.

  4. This will probably give them godlike powers over whoever doesn’t control ASI.

  5. In general there’s a lot we don’t understand about modern deep learning. Modern AIs are trained, not built/​programmed. We can theorize that e.g. they are genuinely robustly helpful and honest instead of e.g. just biding their time, but we can’t check.

  6. Currently no one knows how to control ASI. If one of our training runs turns out to work way better than we expect, we’d have a rogue ASI on our hands. Hopefully it would have internalized enough human ethics that things would be OK.

  7. There are some reasons to be hopeful about that, but also some reasons to be pessimistic, and the literature on this topic is small and pre-paradigmatic.

  8. Our current best plan, championed by the people winning the race to AGI, is to use each generation of AI systems to figure out how to align and control the next generation.

  9. This plan might work but skepticism is warranted on many levels.

  10. For one thing, there is an ongoing race to AGI, with multiple megacorporations participating, and only a small fraction of their compute and labor is going towards alignment & control research. One worries that they aren’t taking this seriously enough.

• Daniel Kokotajlo 6 Jun 2023 18:38 UTC

  76 points

  55

  on: Trans­for­ma­tive AGI by 2043 is <1% likely

  Thanks for this well-researched and thorough argument! I think I have a bunch of disagreements, but my main one is that it really doesn’t seem like AGI will require 8-10 OOMs more inference compute than GPT-4. I am not at all convinced by your argument that it would require that much compute to accurately simulate the human brain. Maybe it would, but we aren’t trying to accurately simulate a human brain, we are trying to learn circuitry that is just as capable.
  
  Also: Could you, for posterity, list some capabilities that you are highly confident no AI system will have by 2030? Ideally capabilities that come prior to a point-of-no-return so it’s not too late to act by the time we see those capabilities.