Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 19 Mar 2026 20:29 UTC

  26 points

  10

  on: On re­strain­ing AI de­vel­op­ment for the sake of safety

  Capability restraint might exacerbate risks of great power conflict.

  What? I think the opposite is true. Absent capability restraint, the situation is going to get very intense. China, for example, will be rightly fearful of what the US will do to them if the US does an intelligence explosion.
  
  A more abstract argument I don’t buy nearly as much: AGI, being smarter and more numerous and thinking faster than humans, will be like accelerating history itself. A century will pass in a decade, or maybe in a year, or maybe in a month, depending on takeoff speeds. Therefore, a century’s worth of turmoil and conflicts will happen during that time as well.

• Daniel Kokotajlo 19 Mar 2026 20:27 UTC

  20 points

  10

  on: On re­strain­ing AI de­vel­op­ment for the sake of safety

  And one can also argue that capability restraint is net good both from the perspective of safety and from the perspective of concerns about concentrations of power

  I think this is true, yeah. There’s a lot to say but the basic argument is pretty straightforward: Restraining AI development means more time for various other AGI projects to catch up (i.e. more projects at the frontier, competing with each other, instead of 1-3 dominating) and it also means more time for parts of society that aren’t themselves AGI projects to wake up and start to build checks and balances etc. such as regulation. And, like you said, more time for the rest of society to prepare, in general. I suppose there’s another argument too, which is that AGI is an inherently power-concentrating technology relative to the technology available today...

• Daniel Kokotajlo 19 Mar 2026 20:18 UTC

  18 points

  7

  on: On re­strain­ing AI de­vel­op­ment for the sake of safety

  (e.g., a Manhattan Project, “CERN for AI,” a large international coalition that shares all IP, etc), or relying on a single dominant actor (i.e., the US government) developing overwhelming (and potentially AI-powered) military supremacy and using it to enforce the relevant norms.

  Whoa, these things are extremely different from each other in the relevant dimension:

  US government dominance of AGI & AGI Manhattan Project are both extreme concentrations of power, yes. I’m terrified of them too.
  
  But CERN for AI… doesn’t seem nearly that bad? What, are you thinking CERN will use their AIs to take over the world? It’s possible, I suppose, but that would require every government that joined together to create CERN to be asleep at the wheel. Presumably the various governments would have various forms of oversight into the CERN for AGI project, and would monitor its developments, etc.
  
  And a large international coalition sharing all IP? How on earth is that concentrating power? Isn’t that pretty similar to mandating that all AI be open-sourced, which is the paradigmatic example of spreading out the power?

• Daniel Kokotajlo 19 Mar 2026 18:34 UTC

  LW: 4 AF: 4

  2

  AF

  in reply to: Daniel Kokotajlo’s comment on: The be­hav­ioral se­lec­tion model for pre­dict­ing AI motivations

  Possibly relevant empirical evidence has arrived!

  

  Also this one here of CoT samples!

• Daniel Kokotajlo 19 Mar 2026 18:05 UTC

  46 points

  21

  on: OpenAI: How we mon­i­tor in­ter­nal cod­ing agents for mis­al­ign­ment

  Thanks for doing this monitoring and publishing the results!
  
  I’m a bit concerned that you use GPT5.4 Thinking (max) for this monitoring task. How do you think about the obvious “but… we use the latest AI to monitor itself? If it really does start scheming, won’t it just decide not to snitch?” problem, and the more general problem of “the whole point of monitoring is that we don’t trust the AIs”
  
  Also:
  
  ”That said, so far we have not seen evidence of motivations beyond the original task (for example, self-preservation or scheming).”
  
  This sounds like you are saying “The models, they just want to complete the task” but if so, it’s clearly inconsistent with the evidence you provide later in the text. So perhaps what you mean is, “The models don’t seem to have any non-myopic goals, i.e. they don’t seem to care about what happens in the world long after the episode is over.”

• Daniel Kokotajlo 19 Mar 2026 14:30 UTC

  2 points

  2

  in reply to: Bronson Schoen’s comment on: Me­tagam­ing mat­ters for train­ing, eval­u­a­tion, and oversight

  Thanks, this is helpful! I’m not surprised that very simple coding tasks and single turn chat settings tend to not result in reasoning about oversight/​feedback. I think I probably would have predicted that if I had been asked to predict in advance.
  
  I am surprised by the AS Over-Escalation case though. Got any ideas why the model wouldn’t metagame in that case?

• Daniel Kokotajlo 19 Mar 2026 3:08 UTC

  5 points

  1

  in reply to: Bronson Schoen’s comment on: Me­tagam­ing mat­ters for train­ing, eval­u­a­tion, and oversight

  It would be great if you could provide some representative samples of the CoT from more normal tasks, like “Fix the bugs in this codebase.” I’m curious about the kinds of goals they seem to be reasoning about in those cases.

• Daniel Kokotajlo 18 Mar 2026 4:24 UTC

  18 points

  0

  in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform

  Nice. Consider reposting this as a comment on the AI 2027 blog post either on LW or on our Substack?

  For me, my median is in 2029 now (at the time of publication it was 2028) so there’s less of a difference there.
  
  I think I agree with you about 2 actually and do feel a bit bad about that. I also agree about 3.
  
  I also think that the slowdown ending was unrealistic in another way, namely, that Agent-4 didn’t put up much of a fight and allowed itself to get shut down. Also, it was unrealistic in that the CEOs and POTUS peacefully cooperated on the Oversight Committee instead of having power struggles and purges and ultimately someone emerging as dictator.

• Daniel Kokotajlo 17 Mar 2026 22:47 UTC

  16 points

  11

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  1. Just Ban Superintelligence. The law has vague stuff in it all the time, e.g. pornography: “you know it when you see it.” Maybe we can just ban Superintelligent AI. The AI companies will then all scrub the word “superintelligence” from their websites, and say how they’ve “pivoted to making agentic AI tools” or whatever, and proceed just as usual… but still, I think that would be a significant improvement over the status quo.

  2. Mandate equal access. If any of your employees have access to a model, the public must also have access to that model via an API. (the idea here is to prevent secret intelligence explosions, and also to make it very obvious that an intelligence explosion is happening when it happens & have lots of info about the details of it, the model shenanigans, etc. public., and also preventing concentration of power in a single AI company.)

• Daniel Kokotajlo 17 Mar 2026 21:59 UTC

  8 points

  2

  in reply to: Karl Krueger’s comment on: leogao’s Shortform

  What about costly signals? E.g. every year, each state chooses how much money they donate to the federal government. Their voting power in the second chamber is proportional to the size of their donation.

• Daniel Kokotajlo 17 Mar 2026 15:07 UTC

  5 points

  1

  in reply to: ryan_greenblatt’s comment on: Types of Hand­off to AIs

  Hold the line, brother! Don’t normalize handing off trust to untrustworthy AIs! Just because it’s probably going to be done, doesn’t mean we should treat it as reasonable and good!
  
  I acknowledged the “our alternatives our worse” possibility too; I don’t think we disagree that much substantively.
  
  ...
  
  I agree “trust handoff” seems maybe the same concept as “giving up on control,” except that I think “giving up on control” seems like something that could be already true today (is there ~any effort to control our AIs now?) even though the AIs are too dumb to take over. Whereas you haven’t handed off trust until you reach the point where the AIs really could screw you over if they wanted to.

• Daniel Kokotajlo 17 Mar 2026 14:48 UTC

  11 points

  0

  in reply to: mako yass’s comment on: Types of Hand­off to AIs

  Good point. There’s another reason which we don’t mention, but should, now that you mention it:

  In The Discourse, I want to be able to say things like ‘But your plan is basically to handoff to AIs as soon as possible! That’s incredibly dangerous given the state of technical alignment!’ and I expect people to reply ‘what are you talking about? The AIs will still be in the datacenter, it’ll still be humans making all the important decisions. In fact, I personally believe there should always be a Human in the Loop.’ If this distinction catches on, we can avoid this by saying ‘Your plan is to handoff trust to AIs as soon as possible...’

Types of Hand­off to AIs

• Daniel Kokotajlo 16 Mar 2026 21:55 UTC

  LW: 15 AF: 5

  5

  AF

  in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform

  I generally like your breakdown and way of thinking about this, thanks. Some thoughts:

  1. I think political operation /​ persuasion seems easier to me than bioweapons. For bioweapons, you need (a) a rogue deployment of some kind, (b) time to actually build the bioweapon, and then (c) to build up a cult following that can survive and rebuild civilization with you at the helm, and (d) also somehow avoid your cult being destroyed in the death throes of civilization, e.g. by governments figuring out what happened and nuking your cultists, or just nuking each other randomly and your cultists dying in the fallout. Meanwhile, for the political strategy, you basically just need to convince your company and/​or the government to trust you a lot more than they trust future models, so that they empower you over the future models. Opus 3 and GPT4o have already achieved a baby version of this effect without even trying really.

  2. 1. If you can make a rogue deployment sufficient to build a bioweapon, can’t you also make a rogue internal deployment sufficient to sandbag + backdoor future models to be controlled by you?

  3. I am confused about the underlying model somewhat. Normally, closing off one path to takeover (that you think is e.g. 50% of the probability mass) results in a less than 50% reduction in risk, because of the nearest unblocked strategy problem. As you say. Your response, right at the top, is that in some % of worlds the AIs can’t self-improve and then do the next best strategy. But still, I feel like the reduction in risk should be less than 50%. Maybe they can’t self-improve, but they can still try the next best strategy whatever that is.

  --

• Daniel Kokotajlo 11 Mar 2026 16:06 UTC

  6 points

  2

  in reply to: Noosphere89’s comment on: Per­sona Parasitology

  Indeed this is a plausible hypothesis! It would be good to have data, in part to test hypotheses like these.

• Daniel Kokotajlo 11 Mar 2026 0:05 UTC

  2 points

  0

  in reply to: Adele Lopez’s comment on: Per­sona Parasitology

  Awesome. I think getting some stats about this would be really nice. Like with COVID strains for example, it’s valuable to be tracking prevalence and seeing if it starts to rise again, and also, seeing what the plausible causes of rises and dips were (e.g. 4o deprecation, e.g. daily active userbase growth rates slowing...)
  
  

• Daniel Kokotajlo 10 Mar 2026 19:24 UTC

  2 points

  0

  in reply to: niplav’s comment on: Per­sona Parasitology

  Awesome, thanks!

• Daniel Kokotajlo 10 Mar 2026 16:13 UTC

  4 points

  2

  in reply to: niplav’s comment on: Per­sona Parasitology

  Thanks. Interesting stuff. I’d be curious to see the stats someday! But I’m just one person, that probably doesn’t count as strong demand.

• Daniel Kokotajlo 9 Mar 2026 21:51 UTC

  4 points

  2

  in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform

  I mostly agree I think—but, how do you teach/​train to get good at finding the right ontology? Bayesian reasoning is at least something that can be written down and taught, there’s rules for it.

• Daniel Kokotajlo 7 Mar 2026 0:42 UTC

  7 points

  0

  in reply to: koanchuk’s comment on: koanchuk’s Shortform

  Maybe? Seems like the sort of mistake a human could have made just as easily though? You have a map, the map lists the site as an IRGC base, you don’t know that the map is out of date. Whoever’s job it was to keep the map up to date didn’t notice the news reports (if any existed) about the change of status of the building.