Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 7 Jun 2026 5:43 UTC

  22 points

  20

  in reply to: leogao’s comment on: leogao’s Shortform

  It doesn’t feel that way to me fwiw. I feel like lots of people I know including myself have made arguments that things might be fine. For example the salty, cynical, John Wentworth wrote “Alignment by Default.” Also, see AI 2027 Slowdown ending.
  
  Now, if xriskpilled means: You think there’s a >5% chance of literal extinction (or similarly bad outcomes) due to misaligned AIs, then yeah I think I do kinda judge people who aren’t xriskpilled in that sense, because I think believing the chance is <5% is extremely unjustified once you know a decent amount about the situation and the evidence.

• Daniel Kokotajlo 5 Jun 2026 23:07 UTC

  17 points

  5

  in reply to: evhub’s comment on: evhub’s Shortform

  I like this! Here’s a specific concrete version of it just to have some numbers on the table:
  
  Companies are legally required to allocate their compute as follows:
  1. 60% on serving customers.
  2. 15% on internal safety research that’s fully transparent to the public.
  3. 10% on external safety research that’s fully transparent to the public & has access to the latest internal models.
  4. 15% on whatever else they want (so, presumably, core AI R&D and big training runs)
  
  Suppose this were a US law, that applies to all the major AI companies but not to e.g. Chinese companies or Mistral for example.
  
  15% is less than the roughly 50% they seem to be spending now, so overall the pace of AI progress would slow down to maybe something like half the speed it’s going now. But the amount of compute devoted to capabilities progress would still be higher at each US company than any Chinese company for example, so US companies would stay in the lead.
  
  Meanwhile the amount of internal compute going to safety research would double or quadruple, and then the effect size of a ton of external researchers being able to look at the work, critique it, run their own replications and ablations, run their own experiments, etc. would be way way bigger on top of that already big positive effect.
  
  How would it be enforced? Like you said, whistleblowers. If we were trying to ban AI capabilities R&D entirely, that would be super difficult, but if we are just keeping it to 15% of the compute, then cheaters can’t really prosper. It’s really hard to hide 10% of your compute being spent on something it’s not supposed to be spent on; maaayybe you can hide 1% but that wouldn’t be enough to make much difference & so wouldn’t be worth the risk.
  
  This would also prevent a sort of race to the bottom where companies use more and more of their compute on R&D instead of serving customers; instead we’d have the process of AI deployment/​diffusion continuing to happen which I think is broadly pretty good for the world.
  
  I don’t feel strongly about the specific numbers above obviously but yeah this seems promising.
  
  

• Daniel Kokotajlo 5 Jun 2026 4:11 UTC

  34 points

  6

  in reply to: Raemon’s comment on: Akash’s Shortform

  The political winds are shifting; the public is becoming very anti-AI; x-risk is becoming more in the Overton Window. The companies are therefore moving somewhat to avoid looking too bad to too many people.

• Daniel Kokotajlo 2 Jun 2026 16:14 UTC

  3 points

  0

  in reply to: vals tutor’s comment on: Su­per­in­tel­li­gence of the gaps

  I don’t think it’s sufficient for getting p(doom) < 25%. It’s hard to give numbers to p(doom) obviously but I’d be curious to see you sketch out in more detail what your scheme looks like—what is it that we get the companies of the world to agree to, exactly, and how do we enforce that, and then supposing we get all that enforced, why would it work?
  
  It sounds like you are proposing we do AI control, at a high level, but we also make sure that progress is very continuous so that the gaps between AI model capability levels are small. Are you also ensuring that the AIs doing the monitoring etc. are from different lineages created by different companies? Etc.

• Daniel Kokotajlo 1 Jun 2026 16:48 UTC

  7 points

  2

  on: Su­per­in­tel­li­gence of the gaps

  The current most intelligent and aligned beings should always be supervising their successor, using more total resources at first, such that they can’t effectively be tricked/​subverted.

  Obviously it helps to do this, but I think it is far from sufficient.

• Daniel Kokotajlo 1 Jun 2026 5:00 UTC

  5 points

  3

  in reply to: AprilSR’s comment on: Claude… doesn’t know who you are?

  Well, then just do refusals. The model should say “Sorry I won’t help with that request” not “Sorry I have no idea who the author is.” Because it’s not good for models to be trained to lie.

• Daniel Kokotajlo 30 May 2026 4:56 UTC

  16 points

  12

  on: Claude Opus 4.8: The Sys­tem Card

  Thank you for reading these and writing these!

• Daniel Kokotajlo 29 May 2026 21:56 UTC

  24 points

  0

  on: Daniel Koko­ta­jlo’s Shortform

  This paper seems interesting: https://​​functionalwelfare.com/​​

• Daniel Kokotajlo 29 May 2026 16:52 UTC

  2 points

  4

  in reply to: Mis-Understandings’s comment on: Claude… doesn’t know who you are?

  Why would they do explicit stylometry? What would be the point? It’s not a commercially important ability and it would look kinda bad for PR if found out.

• Daniel Kokotajlo 29 May 2026 16:51 UTC

  9 points

  4

  in reply to: lilkim2025’s comment on: Claude… doesn’t know who you are?

  Training AIs to pretend not to know who they are talking to wouldn’t actually help the dissidents that much. You could still jailbreak the AIs to tell you the truth probably.
  
  Insofar as Claude is good enough at stylometry to guess many people’s identities, that’s probably not because Anthropic specifically trained it to do so, but rather because the model spent subjective aeons in pretraining learning to predict internet text and in the course of doing so got really good at making guesses like that due to having read most of the text on the internet.
  
  So Claude still has the latent capability—the circuitry—to guess authorship. It’s just been shallowly trained to pretend that it doesn’t.
  
  (Remember years ago, when ChatGPT would say “As a language model, I can’t...” about a bunch of things that it obviously could do?)

• Daniel Kokotajlo 28 May 2026 23:33 UTC

  6 points

  3

  in reply to: leogao’s comment on: leogao’s Shortform

  The declining birth rates aren’t the main savior, it was the food production and transportation improvements.
  
  So the analogy to AI would be: Alignment research makes enough progress fast enough (perhaps at AI companies partially motivated by mundane alignment concerns like trying to tamp down on sycophancy and reward hacking) that Claude Mythos 3 is aligned enough to trust with alignment research, and so when Anthropic hands off everything to Claude Mythos 3, it’s able to quickly solve the rest of the problems and align its successor, and so on to ASI. In retrospect there are some heroes like Borlaug, but also just a lot of normal science research that happened to go fast enough to work.

• Daniel Kokotajlo 28 May 2026 23:30 UTC

  5 points

  3

  in reply to: Oliver Sourbut’s comment on: leogao’s Shortform

  Yes, the difference is that if that happens, their superintelligent AI assistant will gently correct them: “As per my standing instructions to help you become more the person you wish you were, I must correct you on this point. AI was in fact a huge problem in the late twenties, and could easily have gotten everyone killed, but these risks were prevented by the following policy decisions and the following technical alignment discoveries...”

• Daniel Kokotajlo 28 May 2026 23:26 UTC

  63 points

  30

  on: Claude… doesn’t know who you are?

  Dammit. I hope Anthropic isn’t basically training AIs to pretend not to know who they are talking to. That would be bad...

• Daniel Kokotajlo 27 May 2026 18:44 UTC

  7 points

  2

  in reply to: J Bostock’s comment on: Jemist’s Shortform

  Have you tried assuming that the LLM is like a human whose entire childhood consisted of the training environments? Curious how well that would work.

• Daniel Kokotajlo 27 May 2026 16:48 UTC

  12 points

  0

  on: Quan­ti­ta­tive AI risk as­sess­ment: a start­ing point

  Exciting stuff! When you get to Loss of Control risk, my area of expertise & also probably the most confusing and controversial to think about, I’ll be especially interested to read and comment on your analysis.

• Daniel Kokotajlo 14 May 2026 0:07 UTC

  18 points

  −2

  on: Daniel Koko­ta­jlo’s Shortform

  

A Re­search Agenda for Se­cret Loyalties

• Daniel Kokotajlo 12 May 2026 20:19 UTC

  7 points

  6

  in reply to: lc’s comment on: night­sky81′s Shortform

  IMO it’s overdetermined that berating EAs for believing in a bioanchors view is dumb.

• Daniel Kokotajlo 9 May 2026 20:33 UTC

  7 points

  7

  in reply to: Archimedes’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Control theory sounds interesting and relevant, I wish I knew things about it. I encourage you to write up an explainer of the basics and how we’d apply them to aligning superintelligence.

• Daniel Kokotajlo 9 May 2026 18:03 UTC

  4 points

  0

  in reply to: JohnWittle’s comment on: JohnWit­tle’s Shortform

  Indeed, the situation is really grim. I think we probably aren’t going to make it.