Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 9 Apr 2026 18:49 UTC

  3 points

  0

  in reply to: Philip Harker’s comment on: Daniel Koko­ta­jlo’s Shortform

  Automated AI R&D is extremely non-desireable from a safety standpoint. I would be way more optimistic about loss-of-control risk, and various other risks as well, if AI R&D automation was somehow effectively banned.

• Daniel Kokotajlo 9 Apr 2026 3:23 UTC

  2 points

  0

  in reply to: Fabien Roger’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yep I basically agree. Main argument still stands, the other ones are weakened to various degrees in 2026 compared to 2022, at least when applied to stealth abilities. Still have nonzero force though.
  
  

• Daniel Kokotajlo 8 Apr 2026 21:25 UTC

  15 points

  7

  in reply to: ryan_greenblatt’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yes please explain thanks!
  
  Here’s a gdoc link to my views on this subject as of years ago. The text:
  
  If a dangerous capability eval doesn’t involve fine-tuning the model to do the dangerous thing, the eval only establishes a lower bound on capability, not an upper bound or even an approximate upper bound. Why? Because:

  1. For future powerful AGI, if it’s misaligned, there’s a good chance it’ll be able to guess that it’s being evaluated (situational awareness) and sandbag /​ pretend to be incapable. The “no sandbagging on checkable tasks” hypothesis depends on fine-tuning. https://​​www.alignmentforum.org/​​posts/​​h7QETH7GMk9HcMnHH/​​the-no-sandbagging-on-checkable-tasks-hypothesis Moreover we don’t know how to draw the line between systems capable of sandbagging and systems incapable of sandbagging.

  2. Moreover, for pre-AGI systems such as current systems, that may or may not be incapable of sandbagging, there’s still the issue that prompting alone often doesn’t bring out their capabilities to the fullest extent. People are constantly coming up with better prompts that unlock new capabilities; there are tons of ‘aged like milk’ blog posts out there where someone says “look at how dumb GPT is, I prompted it to do X and it failed” and then someone else uses a better prompt and it works.

  3. Moreover, the RLHF training labs might do might conceal capabilities. For example we’ve successfully made it hard to get GPT4 to tell people how to build bombs etc., but it totally has that capability, it’s just choosing not to exercise it because we’ve trained it to refuse. If someone finds a new ‘jailbreak’ tomorrow they might be able to get it to tell them how to make bombs again. (Currently GPT-4 explicitly refuses, which is great, but in a future regulatory regime you could imagine a lab training their system to ‘play dumb’ rather than refuse, so that they don’t trigger the DC evals.)

  4. Moreover, even if none of the above were true, it would still be the case that your model being stolen or leaked is a possibility, and then there’ll be bad actors with access to the model. They may not have the compute, data, etc. to train their own model from scratch, but they quite likely have enough of the relevant resources to do a small fine-tuning run. So if you don’t fine-tune as part of your evals, you are in the dark about what such bad actors would be able to accomplish. Also, once we have more powerful models, if the model was misaligned and self-exfiltrated or otherwise gained control of its training process, it could do the fine-tuning itself...

  5. Moreover, even if that doesn’t happen, the public and the government deserve to know what THE LAB is capable of. If a small amount of fine-tuning using known techniques could get the model to cause catastrophe, then the lab is capable of causing catastrophe.

  6. Finally, even if you don’t buy any of the above, there’s a general pattern where GPT-N can do some task with a bit of fine-tuning, and GPT-N+M (for some small M possibly <1) can do it few-shot, and GPT-N+O (for some small O possibly <1) can do it zero-shot. So even if you don’t buy any of the previous reasons to make fine-tuning part of the DC evals, there is at least this: By doing fine-tuning we peer into the near future and see what the next generation or two of models might be capable of.

• Daniel Kokotajlo 8 Apr 2026 20:11 UTC

  12 points

  6

  in reply to: 1a3orn’s comment on: Daniel Koko­ta­jlo’s Shortform

  Thanks. Reasoning aloud...
  
  So Yudkowsky was wrong because he said this would happen “by default” whereas in practice it seems to happen only some of the time rather than most of the time /​ in some contexts/​prompts rather than in most contexts/​prompts?
  
  I guess so yeah. I suppose Yudkowsky could say that by “by default” he didn’t mean “most of the time” but rather “most of the time absent defeaters such as having been trained not to do this.” But maybe that’s a weak defense.
  
  But this doesn’t exactly seem like a damning blow against Yudkowsky.
  
  More generally it seems like Yudkowsky was imagining AIs with more ambitious, longer-horizon goals than current AIs who seem obsessed with being-judged-to-have-completed-the-task-in-front-of-them, or reward, or some other such myopic thing.

• Daniel Kokotajlo 8 Apr 2026 19:17 UTC

  5 points

  4

  in reply to: 1a3orn’s comment on: Daniel Koko­ta­jlo’s Shortform

  Sorta. I don’t actually think it’s symmetric. The status and money gradients both favor Anthropic over Redwood, AIFP, etc. As does the not-being-a-weirdo gradient.

• Daniel Kokotajlo 8 Apr 2026 17:57 UTC

  129 points

  30

  on: Daniel Koko­ta­jlo’s Shortform

  

  I’m starting to get a little worried. In 2022 I was part of initiating and shaping the whole dangerous capabilities evals agenda, and I emphasized repeatedly to practically everyone I met that the whole thing would be worthless or worse if fine-tuning didn’t become standard practice as part of eliciting model capabilities. Now it’s 2026, the models are way smarter, and and hundreds of people are working on evaluating dangerous capabilities… and yet they still basically don’t do fine-tuning at all.
  
  This really really needs to change before it’s too late, which could be any year now. The models are getting increasingly situationally aware. The entire bundle of evals we do is basically just asking them nicely to show us how scary they are; the obvious flaw in this plan is that if they are sufficiently smart and sufficiently scary, they’ll just choose not to show us. Fine tuning is a basic, obvious mitigation to this threat model that makes it harder for models to sandbag. But it’s not a panacea. But we aren’t even doing this basic thing!

• Daniel Kokotajlo 8 Apr 2026 15:31 UTC

  8 points

  8

  in reply to: paulfchristiano’s comment on: Daniel Koko­ta­jlo’s Shortform

  Thanks, that’s helpful and makes sense.

• Daniel Kokotajlo 8 Apr 2026 15:28 UTC

  2 points

  0

  in reply to: paulfchristiano’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yeah I was replying narrowly to the point about Reddit comment threads. Perhaps I should have disclaimed.

• Daniel Kokotajlo 8 Apr 2026 13:00 UTC

  4 points

  2

  in reply to: Wei Dai’s comment on: Daniel Koko­ta­jlo’s Shortform

  I disagree, fake reviews are incredibly common on the internet—something like 30% apparently. I’ve also specifically seen at least one example of a Substack blogger talking about the importance of building datacenters in their rural district, who was clearly AI generated.

• Daniel Kokotajlo 8 Apr 2026 12:57 UTC

  6 points

  4

  in reply to: papetoast’s comment on: Claude Mythos Sys­tem Card Preview

  You don’t fine-tune on the eval as part of the production model release, you just do a small amount of fine-tuning only for purposes of the eval (i.e. a branch off the production model lineage) to overcome any sandbagging that might be occurring.

• Daniel Kokotajlo 8 Apr 2026 12:55 UTC

  5 points

  0

  in reply to: Sheikh Abdur Raheem Ali’s comment on: Daniel Koko­ta­jlo’s Shortform

  I didn’t claim they were faked without strong evidence, I said I was worried that a bunch of them might be.
  
  Anyhow, according to a brief Claude search, fake reviews are incredibly common. E.g. https://​​capitaloneshopping.com/​​research/​​fake-review-statistics/​​ says that an average of 30% of reviews are fake. So yeah, numerous companies must be indulging in this practice. And better AI makes it easier.
  
  See also https://​​x.com/​​TheMidasProj/​​status/​​2041614395583664225 which seems to be OpenAI-linked, and https://​​doublespeed.ai/​​.
  
  I think it’s a totally reasonable hypothesis to entertain, which is why I try to ask people I actually know about things like this (who tell me Codex is a bit better in some ways, Claude in others, overall similar) rather than trusting anonymous internet comments.

• Daniel Kokotajlo 8 Apr 2026 12:44 UTC

  9 points

  0

  in reply to: Buck’s comment on: Daniel Koko­ta­jlo’s Shortform

  OK, I’ll stop.

• Daniel Kokotajlo 8 Apr 2026 5:41 UTC

  35 points

  25

  in reply to: Oliver Daniels’s comment on: Daniel Koko­ta­jlo’s Shortform

  Who predicted that models of this capability level would be constantly trying to kill us? I certainly didn’t. I think we have evidence to update away from the extreme optimism tail but not the extreme pessimism tail.

• Daniel Kokotajlo 8 Apr 2026 5:11 UTC

  17 points

  0

  on: Claude Mythos Sys­tem Card Preview

  In their Risk Report, they admit that they don’t do fine-tuning as part of their evals. So, if the model was misaligned, it could simply sandbag.

  

• Daniel Kokotajlo 8 Apr 2026 4:20 UTC

  22 points

  9

  in reply to: 1a3orn’s comment on: Daniel Koko­ta­jlo’s Shortform

  People disagree with each other about things all the time, even after lots of evidence has accumulated. This isn’t exactly the first time. People aren’t perfect rationalists.
  
  For my part, well, making the case that the Anthropic Consensus is wrong is not my top priority, I have lots of other things going on, but I’ve written a bunch about my views on alignment e.g. in AI 2027 and other work. I’d love it if Anthropic made the case for the Anthropic Consensus, in public; I could then write a blog post picking it apart. I’m happy that they are moving in this direction, at least, by writing more content in system cards and risk reports.
  
  

• Daniel Kokotajlo 8 Apr 2026 4:09 UTC

  5 points

  0

  in reply to: paulfchristiano’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yeah, that’s what I hear from people I actually know. I think people may have been under the mistaken impression that I think Claude is significantly better than Codex at coding? I never said that.

• Daniel Kokotajlo 8 Apr 2026 0:19 UTC

  18 points

  −2

  in reply to: Wei Dai’s comment on: Daniel Koko­ta­jlo’s Shortform

  I’ve also seen those comments but I’m worried that a bunch of them might be bots. AI-powered astroturfing seems to be a thing already in political landscape and OpenAI specifically seems to be behind some of it; I wouldn’t put it past them to pay for fake reviews like this.

• Daniel Kokotajlo 7 Apr 2026 23:16 UTC

  77 points

  42

  in reply to: Eric Neyman’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yeah I think the Anthropic Consensus is disastrously false and is going to lead to misaligned Claude takeover.
  
  If I agreed with it, I’m not sure what I’d do if I were in charge of Anthropic, probably I’d still do a bunch of different things, especially costly signals of genuine trustworthiness, but I admit I’d be a lot closer to their (predicted) behavior than I would in reality.

  What links here?

  • StanislavKrym's comment on An easy co­or­di­na­tion prob­lem? by KatjaGrace (8 Apr 2026 7:22 UTC; 3 points)

• Daniel Kokotajlo 7 Apr 2026 23:13 UTC

  6 points

  2

  in reply to: Vladimir_Nesov’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yes, I agree with that.

• Daniel Kokotajlo 7 Apr 2026 23:12 UTC

  15 points

  2

  in reply to: roha’s comment on: Daniel Koko­ta­jlo’s Shortform

  TBC I don’t think they have a big lead. I just mean they just barely overtook OpenAI in my estimation. A major indicator is that they seem to have caught up, or even slightly surpassed, on revenue run rate. Also, they’ve done everything with less compute and fewer employees than OpenAI, meaning they’ve accomplished as much or more, with less, meaning they have an inherent quality/​taste/​talent advantage. There are various other things as well, e.g. Mythos. (But we’ll see how good Spud is.)