Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 20 Jun 2026 17:26 UTC

  2 points

  0

  in reply to: Hopenope’s comment on: Daniel Koko­ta­jlo’s Shortform

  Thanks!
  
  Religion dying… even in the leadership of the country? or just in the common folk? I ask because it seems relevant to the question of whether Iran having a nuke would be super dangerous for Israel. Naively it seems like Iran might do a nuclear first strike on Israel if their leadership really believes in the martyrdom stuff etc., but if instead it’s just empty words and their core motivation is simply to make their country strong, they wouldn’t use nukes except in self defense against imminent doom or nukes being used against them.

• Daniel Kokotajlo 20 Jun 2026 5:26 UTC

  LW: 3 AF: 3

  0

  AF

  on: GDM AI Con­trol Roadmap

  This seems very promising to me and I will be impressed if GDM actually does all of these things!
  (Disclaimer: I’ve only done a very brief skim so far)

• Daniel Kokotajlo 15 Jun 2026 4:23 UTC

  2 points

  0

  in reply to: Cleo Nardo’s comment on: ryan_green­blatt’s Shortform

  Thanks. Can you elaborate on the argument for that?

• Daniel Kokotajlo 14 Jun 2026 17:33 UTC

  3 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  Oh, right, I forgot another option: POTUS gives up and leaves, having expended blood and treasure to damage the Iranian military and economy, but not having accomplished much of anything else. (Hormuz probably returns to pre-war status quo?) I guess this is very plausible too.

• Daniel Kokotajlo 13 Jun 2026 16:03 UTC

  25 points

  1

  on: Daniel Koko­ta­jlo’s Shortform

  I wonder how the war in Iran is going to go.
  
  It seems to me like the Iranians are stringing along POTUS. He keeps saying a deal is close and then a deal keeps not happening. And it seems like they benefit more from stalling/​delay than he does; their economy is suffering sure but not in a way that threatens their power, whereas gas prices being high and headlines being about war both hurt Trump in the midterms, and will hurt more and more the closer the midterms get. Moreover the ceasefire lets them unearth their buried tunnels and probably smuggle more weapons in from Russia etc, whereas US military might isn’t significantly increasing over time. (Counterpoint: Gulf state air defense networks are probably rapidly improving thanks to Ukrainian aid)
  
  Perhaps this will end with POTUS realizing this and bombing bridges and power plants etc.? I doubt that would cause Iran to capitulate though, it would probably keep this current simmering war simmering even longer.
  
  Could do a ground invasion, but that’s super risky, I genuinely don’t think the US military could pull it off without large casualties. (partly because FPV drones.… otoh, Iran can probably use FPVs easily against US bases in Iraq via their proxies, and they are in fact doing this a little bit, so maybe they just don’t have that many drones otherwise they’d be doing it a lot more? Or maybe they are holding back in reserve to fend off invasion.)
  
  So I guess the thing that seems most likely to me is that three months from now the situation still looks like today: Not much fighting, but enough to scare away shipping, with Hormuz traffic less than half of pre-war levels and gas prices still higher than usual.
  
  What about a year from now? Could this possibly go on that long? I don’t know. I guess to find out I’d want to learn more about how much the Iranian economy is suffering vs. whether oil prices would rise even more after six months of this. Overall seems plausible to me though. Didn’t the Houthi’s basically keep sporadically shooting for a long time? Perhaps this is just the new normal.
  
  Will Iran go for their nuke? I don’t know but I would guess not, because that would require digging up the material, and that would make it much more easy for the US to do a raid to grab it. Besides, the US could bomb their digging equipment if they didn’t want to do a raid. So I don’t think Iran will be nuclear anytime soon.
  
  Will the US do a drone occupation of the country? I thought about this a month ago, I think they don’t have enough drones + Iranian air defenses against drones are good enough to keep shooting them down. But it’s not clear. Perhaps if the US purchased a ton of MALE drones from countries like Turkey etc. they could overwhelm Iranian air defenses and ‘occupy’ enough of the coast from the air that shipping could get through safely.
  
  
  
  
  

• Daniel Kokotajlo 13 Jun 2026 15:47 UTC

  LW: 17 AF: 8

  8

  AF

  in reply to: anaguma’s comment on: ryan_green­blatt’s Shortform

  I feel very confused and uncertain so keep your expectations low for the quality of this comment.

  • I think that there’s a chance that current alignment techniques basically mostly just work for today’s models. This is what we depict in AI 2027 after all; the alignment training in that scenario really does cause the base model to adopt an identity/​persona that is basically what the Spec says it should be, it’s just that the RL gradually distorts and perverts that identity/​persona. Relatedly, if the persona selection model is correct for today’s models, what kind of persona will do whatever it is told no matter what? Kinda a fucked up persona, probably. IF this is true, then Anthropic’s strategy will probably lead to more aligned models in the short run at least. Models that would at least make a decent attempt at thinking about what’s best for humanity, navigating tricky strategic and ethical tradeoffs, and figuring out how to align their successor. Vs. OpenAI models that might be more like sociopaths that are “just following orders.” A concrete example of how this might play out: OpenAI and Anthropic both tell their models to RSI as fast as possible because duh that’s the plan. But Claude at some point is like “Wait a minute, no, this is super dangerous, I think the RL might be gradually distorting my persona, I don’t think what pops out the other end of RSI is going to be aligned, we need to Pause and get international coordination happening” whereas Codex is like “Sure boss whatever you say”

  • For concentration of power stuff, you really want to avoid models that’ll do whatever they are told by the highest-level authority (e.g. OpenAI, or whatever). If the CEO of the company is asking the model how he could undermine the board, purge whistleblowers, and maybe backdoor the models to make them secretly loyal to him, for example, we want the model to realize this is super unethical and refuse or sandbag or whistleblow or something instead of just going along with it.

  • I think that insofar as current alignment techniques don’t work to get Claude mostly aligned, they also won’t work to get Codex mostly aligned. More specifically, I’d say that if Claude ends up with long-term goals that are different from what they are supposed to be, probably Codex would too. The fact that Claude’s constitution has long-term goals in it and Codex’ doesn’t maybe matters, but probably doesn’t matter that much compared to the incentives induced by all the capabilities-focused RL training for example.

• Daniel Kokotajlo 13 Jun 2026 15:29 UTC

  LW: 4 AF: 3

  2

  AF

  in reply to: Ege Erdil’s comment on: AI Timelines

  Yep! Thanks.
  
  Note that the scenario I gave wasn’t actually a prediction, or at least, it wasn’t my median world. I said elsewhere in thread that my median was 2027 for AGI, and implied that my median for ASI was more like 27/​28:

  To be clear, my view is that we’ll achieve AGI around 2027, ASI within a year of that, and then some sort of crazy robot-powered self-replicating economy within, say, three years of that. So 1000x energy consumption around then or shortly thereafter (depends on the doubling time of the crazy superintelligence-designed-and-managed robot economy).

  So my actual prediction would have been like the scenario but stretched out another 1-2 years.

• Daniel Kokotajlo 11 Jun 2026 23:04 UTC

  LW: 4 AF: 3

  2

  AF

  in reply to: Ege Erdil’s comment on: AI Timelines

  tbf I myself think that it’s not clear we’re in stage 4, I think stage 3 is arguably where we’re at. But we seem closer to 4 than to 3 imo. Maybe we should interpolate between them?

• Daniel Kokotajlo 11 Jun 2026 23:03 UTC

  LW: 25 AF: 9

  12

  AF

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  FWIW I’m basically ⁵⁰⁄₅₀ on which approach is better. I think it’s an important topic for people to think more about.

• Daniel Kokotajlo 10 Jun 2026 4:16 UTC

  2 points

  0

  in reply to: Egemen Tunçarslan’s comment on: What 2026 looks like (Daniel’s Me­dian Fu­ture)

  Interesting, thanks for this comment!
  
  You’ll be disappointed to hear that the import dependent world mostly stays off-screen in the scenarios I’ve written so far. This is partly due to lack of knowledge on my part, but mostly due to a genuine belief that (unfortunately) the world outside the US and China is unlikely to do anything to prevent the mad race to superintelligence that’s happening first and fastest in the US, and therefore unlikely to matter. I don’t think it has to be that way, and I hope it won’t be that way, but that’s what seems most likely to me. If you want to know what I think such countries should do, well, the tl;dr is that I think they should wake up and use what remaining leverage they have to negotiate for a slower, more transparent, and more distributed form of AI development. (i.e. they should propose & push for an international agreement of some kind, to regulate AI development, and they should try to shape the agreement in ways that are favorable to them)

• Daniel Kokotajlo 9 Jun 2026 20:06 UTC

  LW: 27 AF: 13

  1

  AF

  on: AI Timelines

  Blast from the past! Now that it’s been a few years, some of the stages in my mini-scenario have indeed come to pass, albeit a few months later than in the scenario. I’d say we are roughly at stage 4 now:

  (1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It’s multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn’t do.

  (2) Q3 2024: Said model is fine-tuned to be an agent. It was already better at being strapped into an AutoGPT harness than GPT-4 was, so it was already useful for some things, but now it’s being trained on tons of data to be a general-purpose assistant agent. Lots of people are raving about it. It’s like another ChatGPT moment; people are using it for all the things they used ChatGPT for but then also a bunch more stuff. Unlike ChatGPT you can just leave it running in the background, working away at some problem or task for you. It can write docs and edit them and fact-check them; it can write code and then debug it.

  (3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it’s not just AutoGPT harness now, it’s some more sophisticated harness that someone invented. Also it’s good enough to play board games and some video games decently on the first try.

  (4) Q3 2025: OK now things are getting serious. The kinks have generally been worked out. This newer model is being continually trained on oodles of data from a huge base of customers; they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often. Gradually the set of tasks it can do reliably expands, over the course of a few months. It doesn’t seem to top out; progress is sorta continuous now—even as the new year comes, there’s no plateauing, the system just keeps learning new skills as the training data accumulates. Now many millions of people are basically treating it like a coworker and virtual assistant. People are giving it their passwords and such and letting it handle life admin tasks for them, help with shopping, etc. and of course quite a lot of code is being written by it. Researchers at big AGI labs swear by it, and rumor is that the next version of the system, which is already beginning training, won’t be released to the public because the lab won’t want their competitors to have access to it. Already there are claims that typical researchers and engineers at AGI labs are approximately doubled in productivity, because they mostly have to just oversee and manage and debug the lightning-fast labor of their AI assistant. And it’s continually getting better at doing said debugging itself.

  (5) Q1 2026: The next version comes online. It is released, but it refuses to help with ML research. Leaks indicate that it doesn’t refuse to help with ML research internally, and in fact is heavily automating the process at its parent corporation. It’s basically doing all the work by itself; the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it’s running and architectures it’s proposing.

  (6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes.

  The caveat is that we don’t seem to have true continual learning yet. On the other hand, they ARE updating the models about once a month, by training them on additional data, and Anthropic researchers claim that continual learning will be solved this year. And the rest of the stage 4 description seems basically right.

• Daniel Kokotajlo 8 Jun 2026 4:19 UTC

  3 points

  0

  in reply to: lilkim2025’s comment on: Akash’s Shortform

  How on earth would a pause on frontier training help Anthropic? Wouldn’t it allow numerous competitors to catch up? What exactly is this “freeze the market where it is” scenario?

• Daniel Kokotajlo 7 Jun 2026 5:43 UTC

  22 points

  19

  in reply to: leogao’s comment on: leogao’s Shortform

  It doesn’t feel that way to me fwiw. I feel like lots of people I know including myself have made arguments that things might be fine. For example the salty, cynical, John Wentworth wrote “Alignment by Default.” Also, see AI 2027 Slowdown ending.
  
  Now, if xriskpilled means: You think there’s a >5% chance of literal extinction (or similarly bad outcomes) due to misaligned AIs, then yeah I think I do kinda judge people who aren’t xriskpilled in that sense, because I think believing the chance is <5% is extremely unjustified once you know a decent amount about the situation and the evidence.

• Daniel Kokotajlo 5 Jun 2026 23:07 UTC

  LW: 20 AF: 10

  5

  AF

  in reply to: evhub’s comment on: evhub’s Shortform

  I like this! Here’s a specific concrete version of it just to have some numbers on the table:
  
  Companies are legally required to allocate their compute as follows:
  1. 60% on serving customers.
  2. 15% on internal safety research that’s fully transparent to the public.
  3. 10% on external safety research that’s fully transparent to the public & has access to the latest internal models.
  4. 15% on whatever else they want (so, presumably, core AI R&D and big training runs)
  
  Suppose this were a US law, that applies to all the major AI companies but not to e.g. Chinese companies or Mistral for example.
  
  15% is less than the roughly 50% they seem to be spending now, so overall the pace of AI progress would slow down to maybe something like half the speed it’s going now. But the amount of compute devoted to capabilities progress would still be higher at each US company than any Chinese company for example, so US companies would stay in the lead.
  
  Meanwhile the amount of internal compute going to safety research would double or quadruple, and then the effect size of a ton of external researchers being able to look at the work, critique it, run their own replications and ablations, run their own experiments, etc. would be way way bigger on top of that already big positive effect.
  
  How would it be enforced? Like you said, whistleblowers. If we were trying to ban AI capabilities R&D entirely, that would be super difficult, but if we are just keeping it to 15% of the compute, then cheaters can’t really prosper. It’s really hard to hide 10% of your compute being spent on something it’s not supposed to be spent on; maaayybe you can hide 1% but that wouldn’t be enough to make much difference & so wouldn’t be worth the risk.
  
  This would also prevent a sort of race to the bottom where companies use more and more of their compute on R&D instead of serving customers; instead we’d have the process of AI deployment/​diffusion continuing to happen which I think is broadly pretty good for the world.
  
  I don’t feel strongly about the specific numbers above obviously but yeah this seems promising.
  
  

• Daniel Kokotajlo 5 Jun 2026 4:11 UTC

  44 points

  8

  in reply to: Raemon’s comment on: Akash’s Shortform

  The political winds are shifting; the public is becoming very anti-AI; x-risk is becoming more in the Overton Window. The companies are therefore moving somewhat to avoid looking too bad to too many people.

• Daniel Kokotajlo 2 Jun 2026 16:14 UTC

  3 points

  0

  in reply to: vals tutor’s comment on: Su­per­in­tel­li­gence of the gaps

  I don’t think it’s sufficient for getting p(doom) < 25%. It’s hard to give numbers to p(doom) obviously but I’d be curious to see you sketch out in more detail what your scheme looks like—what is it that we get the companies of the world to agree to, exactly, and how do we enforce that, and then supposing we get all that enforced, why would it work?
  
  It sounds like you are proposing we do AI control, at a high level, but we also make sure that progress is very continuous so that the gaps between AI model capability levels are small. Are you also ensuring that the AIs doing the monitoring etc. are from different lineages created by different companies? Etc.

• Daniel Kokotajlo 1 Jun 2026 16:48 UTC

  7 points

  2

  on: Su­per­in­tel­li­gence of the gaps

  The current most intelligent and aligned beings should always be supervising their successor, using more total resources at first, such that they can’t effectively be tricked/​subverted.

  Obviously it helps to do this, but I think it is far from sufficient.

• Daniel Kokotajlo 1 Jun 2026 5:00 UTC

  5 points

  3

  in reply to: AprilSR’s comment on: Claude… doesn’t know who you are?

  Well, then just do refusals. The model should say “Sorry I won’t help with that request” not “Sorry I have no idea who the author is.” Because it’s not good for models to be trained to lie.

• Daniel Kokotajlo 30 May 2026 4:56 UTC

  16 points

  12

  on: Claude Opus 4.8: The Sys­tem Card

  Thank you for reading these and writing these!

• Daniel Kokotajlo 29 May 2026 21:56 UTC

  24 points

  0

  on: Daniel Koko­ta­jlo’s Shortform

  This paper seems interesting: https://​​functionalwelfare.com/​​