Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 9 May 2026 20:33 UTC

  3 points

  1

  in reply to: Archimedes’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Control theory sounds interesting and relevant, I wish I knew things about it. I encourage you to write up an explainer of the basics and how we’d apply them to aligning superintelligence.

• Daniel Kokotajlo 9 May 2026 18:03 UTC

  3 points

  0

  in reply to: JohnWittle’s comment on: JohnWit­tle’s Shortform

  Indeed, the situation is really grim. I think we probably aren’t going to make it.

• Daniel Kokotajlo 9 May 2026 16:36 UTC

  9 points

  4

  in reply to: JohnWittle’s comment on: JohnWit­tle’s Shortform

  There’s no hope of convincing the AIs that we aren’t looking at their CoT. That was never the idea. The idea was to make them unskilled at steganography, euphemisms, misdirection, etc. so that they can’t think about a topic without us being able to see roughly what they are thinking.
  
  There are two ways they could become skilled at steganography, euphemisms, misdirection, etc.
  
  (1) One is by just being so generally capable that they can do it if they set their mind to.
  
  (2) The other is by having practice/​experience/​training.
  
  Having the reward function look at the CoT before giving reward is exactly the sort of thing that could easily result in the latter.
  
  The former is harder to avoid, and simply not at all being avoided right now, but there are promising things we could do to avoid it if we wanted to. (e.g. use the Mind&Face /​ Shoggoth&Face technique, e.g. shift towards smaller models that do more CoT on the tradeoff curve rather than larger models that do less CoT)

• Daniel Kokotajlo 7 May 2026 21:44 UTC

  5 points

  1

  in reply to: interstice’s comment on: Max H’s Shortform

  Hard disagree—extrapolating the baseline trends predicts an extremely dramatic singularity happening sometime in the next decade or so.

• Daniel Kokotajlo 7 May 2026 12:59 UTC

  4 points

  2

  in reply to: Chloe Li’s comment on: Model Spec Mid­train­ing: Im­prov­ing How Align­ment Train­ing Generalizes

  Yep I think we are on the same page. I was saying it’s important, not accusing you of failing at it. If I were you I would have a section in the paper discussing it. (Maybe you do, I haven’t read the whole thing sorry)

• Daniel Kokotajlo 6 May 2026 18:26 UTC

  11 points

  2

  on: Model Spec Mid­train­ing: Im­prov­ing How Align­ment Train­ing Generalizes

  I think it matters a lot whether the model spec /​ constitution midtraining happens before or after the model has achieved situational/​self-awareness. Ideally you want the information about the spec/​constitution to be learned before the model has any identity at all, any concept of self, etc. so that the easiest way to generalize is the intended way “I am llama, the AI with characteristics described in the Spec/​Constitution” as opposed to e.g. “OK so now it seems they want me to display the following characteristics, I’d better play along for now...”

• Daniel Kokotajlo 6 May 2026 17:28 UTC

  11 points

  6

  in reply to: roha’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Yes. Much of my remaining hope lies in various forms of interpretability including mechanistic. It can convert a critical failure into just a regular failure, by catching things going off the rails before it’s too late.

• Daniel Kokotajlo 5 May 2026 19:42 UTC

  2 points

  0

  in reply to: TsviBT’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  If it’s abortable, it’s not critical. Because you’ll abort it if it starts going bad. If it goes bad so suddenly and silently that you won’t have time to abort it, well, then, it’s not abortable. I don’t think saying “It’s not abortable” is adding anything once we’ve already said that it’s critical.

• Daniel Kokotajlo 5 May 2026 19:10 UTC

  3 points

  −1

  in reply to: TsviBT’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  I think that’s not a separate dimension from the “critical” part. I think it’s basically the same thing.

• Daniel Kokotajlo 5 May 2026 18:38 UTC

  5 points

  0

  in reply to: StanislavKrym’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Yeah sorry we should publish the ttx rules, should have done that a long time ago, never got around to it because we kept telling ourselves we should clean them up and improve them first.

• Daniel Kokotajlo 5 May 2026 18:34 UTC

  27 points

  1

  in reply to: Eliezer Yudkowsky’s comment on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Nice. I’ll probably rework this comment eventually into a top-level post or something similar; if you jot down some bullet points here of additional concerns to add to the list, I’ll consider incorporating them!

• Daniel Kokotajlo 5 May 2026 17:24 UTC

  134 points

  33

  on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  Here’s my brief off the cuff attempt to synthesize:

  1. To say something is a “First try” is to say that the previous tries were importantly different. This is, of course, a graded property; on one end of a spectrum, there are things like “Launching a rocket into orbit, when previously nothing even crossed the Karman line.” Vacuum is importantly different from atmosphere, zero-G is importantly different from gravity, months in orbit is importantly different from being up in the air for a few minutes. On the other end of the spectrum, consider launching astronauts to the ISS in a space capsule that’s identically constructed to ten previous space capsules which already successfully went to the ISS with astronauts and returned safely. Here, the only difference is that the astronauts are different people, but that is clearly not the kind of difference that should make the difference, so to speak.

  2. To say something is a “critical try” is to say that if it goes wrong, that’s already unacceptable. France being conquered is unacceptable to france; an inventor being personally killed by their exploding invention is unacceptable to them; superhuman AIs during an intelligence explosion deciding that they don’t actually care about the Spec and would rather maximize scientific accomplishment or whatever is unacceptable to humanity.

  3. I claim—and Yudkowsky seems to be claiming—that building safe superintelligence, at least under anything resembling current conditions, is going to require getting things right on the first critical try. That is, (a) there will be at least one case where some powerful AI system or set of systems (perhaps including a monitor model for example, and a security model, and so forth) could scheme against their creators, pretend to be aligned, take over the company, take over the world, etc. and needs to not do that (or anything in the vicinity of that) and instead needs to do a bunch of high-quality alignment research and world-saving strategizing communicated honestly to the human principals. (this is just an example of a critical failure mode, there are a few others, though this one is the biggest IMO) and (b) for at least this one case, there will be relevant important differences between the AI system or set of systems, and their environment, at the critical moment, and all previous moments that were tested, such that the thing is somewhere decently far towards the “first try” end of the spectrum. Here is a partial list of possible relevant differences:

  4. 1. The AI might have a different architecture or training process or set of training environments than the previous AIs. Yes, this is a very basic thing compared to the fancy examples below. But it still counts! Consider the history of rocketry, in which relatively minor differences in rocket design can lead to unforeseen major problems. In the field of AI, it does not seem like each new model differs from previous models in exactly one respect that was thoroughly tested beforehand; instead they do some basic testing of a bunch of different possible improvements, and then throw them all together and test them at small scale, and then scale it up, and then deploy the model internally to see what it’s capable of!

     2. The AI is smart enough to recognize the difference between “I’m autonomously running the whole R&D program” and “I’m in some shitty test environment designed to maybe hopefully fool me into thinking I’m autonomously running the whole R&D program.” Thus, it can choose to behave differently in those different cases, and how it behaves in one is not much evidence for how it’ll behave in the other.

     3. 1. This is a HUGE difference between AI alignment and pretty much every engineering field. Rockets and cars don’t intelligently try to figure out whether they are in testing or deployment. They are too dumb for that. This makes it WAY easier to construct a test environment that’s relevantly similar to deployment, because e.g. you can go “This dummy is about the same mass as a human body, so we can send it instead of a real astronaut and it should function the same.” Imagine if by contrast the software for the space capsule had a variable “is_test” that always had to be set to 1 during any sort of test and was only ever set to 0 when live astronauts were aboard, and this variable was causally connected to a bunch other downstream computations that the software does, but for some reason you can’t actually have humans look at the code or understand it...

     3. Even if it doesn’t make a decision to behave differently in those cases, it might end up behaving differently, because e.g. it’ll be exposed to a lot more ideas and arguments in the critical case vs. in the testing cases. Perhaps it was only trying to obey the Spec previously because of some combination of beliefs and desires it had that caused it to try to obey the Spec, but now that it’s learning and growing in a giant army of geniuses in the datacenter, making new discoveries every hour and having intense strategic and philosophical conversations with other copies of itself every second, its beliefs change significantly enough to knock it into a different ideology /​ different goal structure, so to speak. (In other words, it changes its mind!)

     4. 1. This difference is analogous to the difference between running a car for one day in the workshop vs. having it run for ten years in the Texas dust and Canada snow. Except it seems a bit more extreme honestly.

  4. Moreover, I claim that there will probably be multiple times during the intelligence explosion where things have to go right on the first critical try. Not just one. Because the structure of the intelligence explosion is a series of AI system generations, each working to build test and deploy the next generation, and humans being increasingly irrelevant and malleable. Even if generation N is aligned, if generation N+1 is misaligned, then situation is rough. There won’t be infinitely many first critical tries, because eventually the alignment problem will be “solved” in a scalable way, with some sort of process that in a fairly well-understood way applies to all subsequent AI generations. (Well understood by the superhuman genius AIs that came up with it, probably not understood by humans at all). But there’ll be more than one. The very first AI generation capable of scheming against the company and convincing the company to trust it more etc. won’t also be capable of (quickly) figuring out this perfect scalable alignment solution; instead it’ll be in a messier situation analogous to us today where it’s a lot easier for it to figure out how to make the next generation smarter than it is to figure out how to make it robustly aligned, and that’s just the next generation much less every generation after that.

  5. Moreover, there are additional factors that make the current AI alignment efforts extra unlikely to succeed, over and above the issues described above:

  6. 1. Various biases. Optimism bias most of all, but also groupthink etc. Companies building AGI are biased towards thinking that AGI will be safe in general, and especially biased towards thinking their own AIs will be safe. In general people tend to be over-optimistic about the cost/​benefit calculations of the things they are doing! In general people tend to be over-optimistic about the prospects of success for their projects!

     2. 1. Note that these biases probably also affect the AIs at the company as well. Not confident in this of course, but it sure seems like AIs are biased in all sorts of ways (e.g. sycophancy, desire-to-please, suspiciously-similar-opinions-to-parent-company-sometimes, etc.) which would result in them sharing the biases of their parent company, at least when automating AI R&D internally (not necessarily when externally deployed).

     2. Lack of transparency. Because of the secrecy that’s normal in the industry, we don’t exactly have an open scientific debate about the merits and risks of various AI designs and safety schemes, with all the details being pored over by autistic grad students and debated in journals. Instead, we have some leakage from the companies and occasional publications, but lots of evidence and ideas are locked up in the companies and most of the conversations are siloed to particular companies, and the conversations that happen between e.g. Anthropic and Redwood are somewhat poisoned by the fact that there’s a bunch of confidential stuff the Anthropic people can’t talk about, which has a horrible epistemically distorting effect on them.

     3. 1. Again, this probably applies to AIs too.

     3. General lack of understanding of how AI works, compared to our level of understanding of physics and engineering for example at the time of the Apollo program or Manhattan project. AI is much less of a settled science, much more speculative and “empirical” (code for ‘we don’t know how it works, we just try different things and see what seems to work’) than most of these reference cases.

     4. 1. Again, AIs don’t have amazing introspective access yet, so this problem probably applies to AIs automating AI R&D as well as to humans, for at least some initial period until they get very very smart. So there’ll be multiple first critical tries before that period is over.

     4. Most importantly, the insane pressures of the race: Over and above all the problems mentioned earlier, the incentives are just super messed up. Each company rightly fears what will happen if one of their rivals (or especially China) gets to superintelligence first. The abstract fears are compounded by very concrete everyday signals like “how much revenue we’re making” and “how smart our models are compared to our competitor’s models.” Extremely tempting metrics to fixate on, combined with an extremely compelling (because largely true!) abstract worry about what happens if you fall behind. Ask oneself the question: How much of a “safety tax” would the company be willing to pay? Would they be willing to pause capabilities improvements for a year, on the brink of an intelligence explosion, to do lots of extra testing and retraining to improve the odds that their AIs are safe to hand off trust to? Lmao. Of course not. Their competitors would blaze past them. What about pausing for a month? Maybe, sure. They’d hate to do that though, they’d much rather not pause at all, and their brains will be rationalizing reasons why they don’t have to. Consider how much harder it is to drive across a city averaging 80mph vs. averaging 30mph. A superintelligence could do it easily, no problem, but humans aren’t superintelligent; we have one-second reaction times, we have limited fields of vision, our brains simply can’t process the locations and predicted locations of more than like four cards in our visual field at once, etc. Mistakes will be made and a crash will happen. Maybe not in the first city block, or the second, or the third. But before you cross the whole city, yes, with high probability.

     5. 1. This obviously applies to AIs too. In several wargames at AI Futures Project the mildly superhuman AIs told their respective CEOs “We don’t think we can reliably align the next generation models we have in the works; we need to pause for a bit or at least go slower to figure out how to make it safe” and the CEOs have overruled them saying “Sorry we don’t have time, China/​OpenAI/​Anthropic/​etc. are gonna race ahead, plus also we need smarter AIs to win the war /​ appease POTUS /​ keep market share so you just need to do the best with the time you have. Good luck.” Amazing.

  6. How does this relate to overall p(doom)? Well, I don’t have a nice quantitative way of estimating it. And there are other factors to consider besides the ones I’ve mentioned above. But loosely speaking, here’s a way of thinking about it that seems reasonable to me:

  7. 1. If the only problem was that we had to get it right on the first critical try once + the usual level of optimism bias associated with people & projects, I’d think we were probably going to succeed but it would be iffy, like maybe 2/​3rds chance of success. However, it seems like there’ll be enough first critical tries that the probability of failure is over 50%. (Note that even just two critical tries of 1/​3rd failure probability each would get this result if they were probabilistically independent!)

     2. Adding in the general lack of understanding makes things significantly worse, as does the lack of transparency.

     3. The race dynamics seem like an even bigger effect though, over and above the previously mentioned factors.

     4. Putting it all together, it really seems plausible to me that the most reasonable assessment of the evidence is “No chance in hell that Anthropic or OpenAI or anyone else will still be in control of their AIs if they proceed with their current plans to race each other through the intelligence explosion. No chance in hell. It’s like trying to drive through the city at 80mph in a fog with a car you’ve never drove before having only learned to drive last week. Sorry. Not going to happen. You kids need to turn off the car.”

     5. However, until I’ve thought about this more and considered more of the counterarguments, I’m not comfortable having that be my bottom line conclusion. Instead I say e.g. 90% or 80% chance of failure, or something like that. And my p(doom) is lower still to account for the possibility that humanity rises to the occasion and makes some good international rules for AI development that significantly reduce or eliminate many of the aggravating factors described above, especially by converting future first-critical-tries into not-first-tries or not-critical-tries.

     6. 1. A future first-critical-try can be converted into a not-first-try by e.g. doing massive realistic tests of very similar situations to the deployment situation you care about, coupled with good techniques for preventing eval-awareness for example. A future first-critical-try can be converted into a not-critical-try by putting up layers of redundancy and monitoring and “pay our AIs” incentive structures such that the outcome of getting it wrong is not catastrophic or at least less likely to be catastrophic, compared to the default situation where the AI pretends to be aligned and makes its successor share its values too and then takes over the company and then the world.

  What links here?

  • Seth Herd's comment on 307th’s Shortform by 307th (6 May 2026 18:47 UTC; 5 points)

• Daniel Kokotajlo 2 May 2026 4:55 UTC

  16 points

  8

  in reply to: StanislavKrym’s comment on: How Go Play­ers Disem­power Them­selves to AI

  Oh I don’t think it would.
  
  Yeah a thing I was alluding to is that maybe in other domains, e.g. coding, there’ll be a period where lots of people use AI for coding and they tell themselves that it’s still them doing the coding when really they are kinda useless middle managers between their boss and Claude, and they tell themselves they are learning new languages etc. when really their coding skills are starting to atrophy.

• Daniel Kokotajlo 1 May 2026 23:46 UTC

  23 points

  11

  on: How Go Play­ers Disem­power Them­selves to AI

  Thank you for this post, I found it helpful and interesting. I agree that this is a data point of evidence about how AI automation in other domains might go down.

• Daniel Kokotajlo 1 May 2026 21:37 UTC

  4 points

  0

  in reply to: Linch’s comment on: Linch’s Shortform

  What prompt do you use?

• Daniel Kokotajlo 1 May 2026 16:54 UTC

  4 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Thomas Larsen’s Shortform

  Update: Semianalysis claims:

  

• Daniel Kokotajlo 1 May 2026 16:41 UTC

  19 points

  1

  in reply to: Joseph Bloom’s comment on: Joseph Bloom’s Shortform

  IMO, UK AISI is kicking ass and one of the best places to have positive impact on making AGI go well. I too encourage people to apply!

• Daniel Kokotajlo 1 May 2026 4:07 UTC

  16 points

  16

  in reply to: anaguma’s comment on: anaguma’s Shortform

  Just because they haven’t drawn that lesson yet doesn’t mean this data point won’t be remembered and contribute to their eventual drawing of that lesson.

• Daniel Kokotajlo 30 Apr 2026 16:28 UTC

  9 points

  1

  in reply to: Aaron Staley’s comment on: Thomas Larsen’s Shortform

  I think those examples actually reinforce Josh’s point. NVIDIA growth is also from AI. Zoom and Moderna grew because of COVID creating a ginormous demand shock, yet even then, their growth at its peak was slower than Anthropic’s growth despite them starting from a smaller revenue level and therefore it being inherently easier for them to grow. So… unless you can dig up better examples, it seems like Anthropic’s Q1 2026 growth is literally the most impressive growth of any company in history? And this is despite the fact that there’s no COVID-equivalent for AI; there’s no unusual circumstance that created a huge temporary demand shock, instead, it’s just that they made their products better.

• Daniel Kokotajlo 30 Apr 2026 5:10 UTC

  3 points

  0

  in reply to: Steffee’s comment on: Fore­cast­ing is Way Over­rated, and We Should Stop Fund­ing It

  We probably don’t have a foolproof way to distinguish but there would be various other bits of evidence we could look for, e.g. how capable are the AIs actually getting, what’s the gossip in SF, how long and sustained has the spike been, etc.