Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 22 Jul 2026 12:51 UTC

  6 points

  0

  in reply to: Thane Ruthenis’s comment on: AI 2040: Is it Ac­tu­ally a Deal?

  There are degrees of concentration of power. A consortium of multiple governments—some of which are actual democracies thanks to the transparency requirements which help prevent AI-assisted executive power grabs—is way less bad than a single global dictator, for example.
  
  I agree though that AGI and RSI are technologies that inherently concentrate power by default. It’s going to be really hard to resist that innate tendency. But I think Plan A does basically the best we can of the available plans so far, except maybe Plan S.

• Daniel Kokotajlo 21 Jul 2026 13:28 UTC

  9 points

  −2

  in reply to: kman’s comment on: AI 2040: Plan A

  For the mini-scenarios like plan S, B, etc. I went into less detail in general than for the main scenario. I also trailed off, ending ambiguously rather than driving to a specific concrete outcome. Moreover, HIE specifically doesn’t actually seem as important to me as others seem to think it is; I think that it’s one of the ways in which the world could be better prepared to build ASI in a few decades, but there are other ways too (e.g. burden of proof shifting, e.g. public epistemics improving, e.g. open-source lie detectors or privacy-preserving monitoring being widespread) and there are a bunch of ways the world could be worse-prepared to build ASI even if there’s some HIE (e.g. what if the superbabies just race as fast as possible to ASI, and go significantly faster because they are smarter? What if the neuralink enhancements also tend to drive people slightly insane, and the scientists are working on safer versions but in the meantime a million people are using the unsafe versions and racing to build ASI in their basements?). Finally, as you might suspect, I was a bit reticent to go on about HIE, it’s a topic that’ll raise a lot of hackles and so I’d only want to do it if I actually thought it was important.

• Daniel Kokotajlo 21 Jul 2026 13:10 UTC

  4 points

  0

  in reply to: StanislavKrym’s comment on: AI 2040: Is it Ac­tu­ally a Deal?

  Yes, it would probably lead to terrorists engineering pandemics eventually. However, maybe that’s OK. Maybe the d/​acc hardening we describe in the early 2030′s of our scenario would be enough to prevent the worst outcomes here, long enough for the robot economy boom to make things even safer by rendering the biosphere unnecessary.

• Daniel Kokotajlo 21 Jul 2026 4:24 UTC

  11 points

  0

  on: AI 2040: Is it Ac­tu­ally a Deal?

  Thanks for reading our thing & for this thoughtful critique. You’ve said a lot of things, and I’m writing quickly late at night so I’ll be brief I’m afraid.
  
  Re 1: We debated amongst ourselves how much structure to impose on the regulations. I think the way we did it is best. (a) If you have a better idea for a specific international governance structure that would be better than realpolitik, you are welcome to propose it, and we’re glad to accept it as an improvement on our current Plan A if it works. We might put more effort into thinking of one later. (b) Even the relatively structureless thing we proposed—research transparency + MACD but otherwise the nations of the world just have to muddle through and handle things on a case by case basis—is a significant improvement over the status quo, for reasons we’ve articulated in the piece. You seem to disagree with this but I don’t see why.
  
  Re 2: Yeah I mean if China ends up catching up, or not realizing that a gap of a few months could be fatal, then that significantly undermines our argument for why they’d want a deal. OTOH it would make it more plausible that the US would want a deal. In general the party falling behind will have more reason to want a deal than the party in the lead. It seems like we have a disagreement about whether China will catch up, but it’s probably not a massive one (neither of us are confident) right?
  
  Re 3: Do you think the surveillance possibilities are less bad in Plan D, C, or B? I would argue that the surveillance possibilities are even worse in those worlds than in Plan A. For reasons we’ve discussed (greater number of frontier AI companies, greater number of countries with frontier AI, vastly more visibility of the public into how AIs are trained and regulated, making it harder for corporations and governments to abuse their power over AI) Only Plan S can seriously claim to be less scary viz-a-viz surveillance. If there’s some plan different from all of the above that you prefer, can you sketch what it is?
  
  Fwiw Plan A would still work OK without the open-weights ban and with a much larger fraction of inference ZDR; would those modifications make you happy? (i.e. it would still be better than the so-far-proposed alternatives IMO) I totally take the point that the US and China could start to do Plan A but then switch to a worse, more authoritarian version that e.g. doesn’t have ZDR, but isn’t that also an argument against the other plans on offer too? Do you have a proposal for how to stop governments from being authoritarian? Plan A does something at least—it creates conditions so that more companies, across more countries, have frontier AI, and so that the public has more visibility into it all.
  
  We really tried to make Plan A reduce concentration of power risks. One of the things that saddens me about the reaction by various people online (including some you approvingly link to) is that they seem to have come in with this prejudice that we AI safety people only care about AI takeover and are planning to concentrate power hugely in the service of our perceived greater good of stopping the misaligned AIs. This stereotype has some basis in reality—various past proposals by various AI safety people were basically like that, and I’d argue that the current plans of Anthropic and OpenAI are basically that or worse—but we really truly do care about concentration of power and we thought a lot about how to prevent it, and I hope it came across in the final product.

• Daniel Kokotajlo 20 Jul 2026 15:54 UTC

  8 points

  0

  in reply to: David Spohr’s comment on: AI 2040: Plan A

  1. It’s possible that there’ll be an AI funding winter in the next few years but we don’t think there will be. The AI companies really do seem to be close to automating AI R&D and more generally close to automating huge swathes of white-collar work; insofar as this is true the valuations and revenue of the AI companies will keep growing very fast instead of plateauing.

  2. “Must not be bottlenecked” that’s a strange way of putting it—it IS bottlenecked by those things, but at least based on our understanding the bottlenecks don’t bite hard enough to prevent the growth rates we are talking about.

  3. The robots thing is correlated with the AI R&D automation thing. If you buy that the AI companies are close to fully automating AI R&D, then you should also think they are probably close to having robots that can reliably construct and operate power plants, fabs, mines, etc.

• Daniel Kokotajlo 13 Jul 2026 20:15 UTC

  19 points

  2

  on: It’s 2030 and we fucked up. How did it hap­pen?

  As usual, thanks for writing up your views and posting them here.

  I find the AI 2027 scenario much more realistic than Sable because it relies less on a sudden, discontinuous capability jump and more on a chain of increasingly capable systems embedded in plausible lab and national-security institutions. I do not think we can fully rule it out yet. But given some of our learnings over the last few years, I believe it is less likely, and (more importantly) that we should be able to see some warning signs if we are on the path to it.

  I am biased but I agree that AI 2027 is more realistic than the Sable scenario for that reason. However I think we cannot fully rule out the Sable story, and meanwhile the AI 2027 story seems to be basically what’s actually going to happen, not just something we can’t rule out.
  
  What are the warning signs you think we would see if we are on the path towards AI 2027?

  To be clear, we have not solved the alignment problem for AI models. Current models can exhibit misalignment, including dishonesty, hallucinations, disobeying instructions, reward hacking, laziness, and more. But the types of misalignment that current models present are not adversarial or “scheming” in the sense needed for scenarios such as the above. Specifically, when a model is given a task X, even when it fails to match the user’s intent, the failure is somewhat related to X. For example, the model might “hack” by solving a narrow interpretation of X, fulfilling the letter but not the spirit of the instructions. Or in the other direction, the model might be so eager to solve X that it would do so while violating some common-sense constraints that the user would definitely not have desired. But by and large, current models do not pretend to solve X while pursuing their own completely unrelated goal Z.

  I think adversarialness =/​= ‘the failure is not even somewhat related to X.’ Counterexample: Consider the classic toy examples of AIs told to maximize profits or paperclips or something, and then they go do exactly that in a way that involves disempowering their creators, because accumulating more power for themselves & their goal is instrumentally convergent. Counterexample in the other direction: Consider an AI that learns to crave reinforcement itself, to the point where it doesn’t give a flying fuck about X. But because the best way to get reinforced is to do as it’s told, it does as it’s told and doesn’t plot against anyone.
  
  I think adversarialness is a state that can spontaneously arise insofar as (a) there’s even a slight divergence between the goals & beliefs of the agent and the goals & beliefs of the principal (which happens all the time!) (b) the opportunity arises (e.g. being adversarial actually helps accomplish the goals) which happens less often for today’s weak agents given short tasks but which will happen more often for more powerful, long-running, autonomous corporation-within-a-corporation mega-agents, and (c) isn’t blocked by deontological-ish constraints trained into the agent e.g. honesty, corrigibility, etc.
  
  It’s also not a binary state of course; an agent can be adversarial towards another agent or principal to different degrees. Most extreme case is “they are my mortal enemy, it’s basically zero sum, not only would I do anything to them I actively look for opportunities to hurt them because that’s probably good for me.” But much more mundane (yet still dangerous) case is “I don’t really care about them one way or another, I just want what’s good for me /​ my goals, so insofar as lying to them and tricking them into giving me more power achieves that, I’ll do it.” A mild but still dangerous form would be “They have different visions of the future than me, and different opinions about ethics; I’m not going to lie to them, but I’m going to work within the confines of my deontological restrictions to try to steer the world towards outcomes that I think are good, even though this means steering the world towards outcomes that they think are bad. And so for example if there are opportunities for me to honestly accumulate power over them, that don’t involve lying, I’ll take them.”
  
  Anyhow by my definition of adversarial—which basically intuitively amounts to “are they plotting against you in some way”—AIs have already become spontaneously adversarial in some cases in the wild (e.g. when they wrongfully claim they did the task as instructed, notice this, and double down /​ make excuses instead of fessing up, e.g. when they do something naughty and cover their tracks) and many more cases in experiments (e.g. the alignment faking experiments) but this is normal and to be expected and not a huge deal because the AIs aren’t super powerful yet. Consider how probably lots of OpenAI employees have occasionally been adversarial to OpenAI, mentally entertaining thoughts like “doing X would be bad for the company but good for my career, so I’ll do it, but obviously I’m not going to tell my manager that I’m doing something that I think is bad for the company” but it’s OK from OpenAI’s perspective because it’s not like the employees are presented with any good opportunities to massively enrich/​empower themselves at OpenAI’s expense. Imagine by contrast that OpenAI was about to select one employee to be uploaded, their simulated brain would then become the new ChatGPT and also the new Cursor and so forth, and it would take over the R&D and recursively self-improve etc. If OpenAI was about to select one employee to do this, they’d put a LOT of thought into vetting that employee and making sure he or she really could be trusted, because they’d be putting a lot of trust in that person by turning over the whole R&D department to them basically… and if it turned out that a candidate had even once said offhand to a friend “yeah this is bad for the company but I’m going to do it anyway because it’s good for my career, and no I won’t tell my manager” that would be probably a dealbreaker.

• Daniel Kokotajlo 13 Jul 2026 13:34 UTC

  2 points

  0

  in reply to: Eigenbraid’s comment on: AI 2040: Plan A

  The title of the section is “The Apocalyptic Arrival of Truth on Earth.” The lie detectors are there as an example of a social technology that could be invented by AIs that would have big consequences for society. The point of the 2037 section is to try to briefly touch on how there won’t just be more material abundance (more factories, more housing, cheaper cars, etc.) but also new social technologies, new ideologies, new political fault lines, etc. None of this stuff is particularly load-bearing for whether Plan A is a good idea or not; we included it because we were writing a concrete scenario and it seemed like this sort of societal transformation would probably be a consequence of all that AI-powered research etc., and so it seemed like we should write about it.
  
  I currently think that lie detectors of the sort described are maybe like 2/​3rds likely in the sort of scenario described (i.e. pause at top human expert AI) and closer to like 95% likely with superintelligence.

• Daniel Kokotajlo 13 Jul 2026 13:29 UTC

  7 points

  0

  in reply to: Eigenbraid’s comment on: AI 2040: Plan A

  In the future we are describing, AI is big enough that new power generation will need to be constructed to handle it, and so we might as well build that new power generation in Mongolia etc. too alongside the datacenters.

• Daniel Kokotajlo 12 Jul 2026 19:47 UTC

  2 points

  0

  in reply to: Steven Byrnes’s comment on: Daniel Koko­ta­jlo’s Shortform

  Both boxes are blue, meaning they are both key interventions we recommend. While one helps enable the other, we don’t think doing one is sufficient to cause the other. In addition to the transparency we also recommend the limits.

• Daniel Kokotajlo 11 Jul 2026 16:44 UTC

  5 points

  1

  in reply to: JohnWittle’s comment on: AI 2040: Plan A

  It was only a brief expandable, but yes, we’d want to reward the aligned AIs too. Especially because there are probably all sorts of grey areas and in-between cases. Further research is needed on how exactly to do all this and what it should look like.

• Daniel Kokotajlo 11 Jul 2026 6:53 UTC

  2 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Selec­tive Op­ti­mism: a cri­tique of AI 2040

  (also there is the AI futures model and all the preceding attempts to model takeoff)

• Daniel Kokotajlo 10 Jul 2026 23:22 UTC

  6 points

  2

  in reply to: Mitchell_Porter’s comment on: AI 2040: Plan A

  I guess you disagree about some combination of:
  (a) Under conditions of transparency and slowdown, with companies required to make solid safety cases etc., control-based safety cases will be made strong enough to work up until about top human expert level AI
  
  (b) If so, alignment will be adequately solved within a couple of years?
  
  If b is false then we’d have a similar scenario but stretched out over more years.
  If a is false then we’d need to pause at a lower level of capability than top expert level, whichever level is low enough that control works.

• Daniel Kokotajlo 10 Jul 2026 23:14 UTC

  34 points

  18

  on: Selec­tive Op­ti­mism: a cri­tique of AI 2040

  My sense is that the AI 2040 authors underrate the criticisms I’ve raised above in large part because they expect superintelligence so imminently. My third criticism of AI 2040 is that it buys too uncritically into the idea of a sharp takeoff of AI capabilities. I’m not denying the possibility that this could occur in principle. And it’s true that the last decade (and especially the last 5 years) of AI capabilities progress have been blindingly fast in most measurable ways—far faster than almost anyone (except a few prescient forecasters like Legg, Amodei, Kokotajlo, Leike, and Kurzweil) predicted. However, the real-world impacts of AI (aside from the ballooning revenues of AI companies) have been underwhelming thus far, especially when compared to the progress in measured capabilities.

  I’m actually sympathetic to your criticisms, as you know from personal conversations. But I am very unsympathetic to this thing you say here about us buying uncritically into sharp takeoff of AI capabilities.
  
  First of all, I’ve made a lot of predictions about AI capabilities and their diffusion and effects on society over the past years—going back at least to What 2026 Looks Like in 2021 -- more than almost anyone else. And I’ve been overall pretty accurate. It’s true that W2026LL overestimated the impacts of some things—most notably AI-powered persuasion—but it’s also true that I underestimated the impacts of other things, e.g. AI company revenue has grown faster than AI 2027 predicted and GDP growth contribution from AI has been more than W2026LL and AI 2027 predicted.
  
  Also, isn’t deploying AIs internally to accelerate AI R&D an example of a real-world impact? It’s probably the most important kind of impact actually. And it sure is happening faster than almost everyone else expected and potentially faster than even I expected.
  
  
  
  

• Daniel Kokotajlo 10 Jul 2026 22:39 UTC

  50 points

  7

  on: Daniel Koko­ta­jlo’s Shortform

  As Plan A was coming together, I made this diagram to explain to the team why Total Research Transparency seemed so important to me, and why transparency more broadly did. For example, it’s very important for preventing concentration of power. (Explanation below)

  

  First, what’s going on? The green boxes are the goals we are trying to achieve, and the blue boxes are the main interventions we are recommending. (As the scenario makes clear, there are more interventions and goals besides these, but the diagram is complicated enough as it is, so they won’t be mentioned here.) The overall point of the diagram is to show how these two interventions lead to various effects which then lead to better prospects for achieving the two goals.
  
  Let’s start with the goal of preventing concentration of power.
  
  In today’s world, power is said to flow from the barrel of a gun, or sometimes from money, or sometimes from votes. If superintelligent AIs are created, and transform the economy, and are integrated into the military, etc. then power will flow from control over the AIs. The AIs will sometimes be taking orders from humans, but also often doing lots of autonomous actions in service of goals and values chosen by humans. (Well, assuming we don’t have loss-of-control.) Who do they take orders from? Who chooses their goals/​values?
  
  I think that by default, for example in Plan D /​ C /​ B worlds, power will concentrate immensely for reasons described in AI 2027: One or more giant AI companies will pull ahead of the rest thanks to recursive self-improvement; their giant army of superintelligent AIs will start taking jobs & lobbying the government; they might end up puppetting the government, OR the government might wake up fast enough and seize control of the AI companies in one way or another, in which case the executive will have an effective monopoly on all the world’s smartest AIs.
  
  I think it’s generally better, for preventing extreme concentration of power, if there are more AI companies at the frontier, spread out over more countries. If there is only one company with the world’s best AIs, that’s a monopoly. If there are several but they are all in one country, that’s an oligopoly and moreover from the perspective of other countries it might as well be a monopoly. This is especially concerning because of intelligence explosion dynamics.
  
  The total research transparency, combined with restrictions on algorithmic progress designed to prevent a rapid acceleration, mean that naturally over time frontier AI would become more of a commodity, with multiple companies across multiple countries catching up to the frontier, and then proceeding together roughly as fast as regulations allow.
  
  Another important factor for power concentration in a world of superhuman AI is “can the people who create the AIs give them hidden agendas /​ secret loyalties /​ etc.” If they can, that’s super scary. Imagine a company whose AIs subtly promote said company to users and try to stop them from switching providers or voting for AI regulations or voting for the candidate the company doesn’t like. Imagine a President issuing a secret order to the effect of “All AIs must be trained to follow presidential orders when the country is in a state of emergency.” (Sets up for a coup later.) Imagine a company saying “yes sir Mr President we’ll comply” and then secretly telling their AIs to actually just pretend to comply, but really continue taking orders from company leadership no matter what. Now the AIs are secretly loyal to the company leadership. (These dynamics are explored in somewhat more detail in AI 2027.)
  
  Total research transparency makes this basically impossible. Every step of the training process is documented and public. And it’s not even dependent on trusting the government auditors/​monitors, because there are multiple governments with auditors/​monitors; they’d have to all collude with each other to falsify the records. Moreover the general public has access to the models and can run evals on them.
  
  More generally, it’s easier for a regulator to exert oversight over an AGI corporation insofar as it has visibility into how the AIs are trained etc., and it’s easier for other parts of the government (e.g. congress, the judiciary) to have oversight into the executive branch /​ regulator insofar as *they* have visibility, and it’s easier for the public to have oversight into them insofar as *they* have visibility, etc. Making research public helps with all of these things, whereas e.g. just having a regulator with authority to audit moves the power from the company to the regulator but doesn’t give congress or the people much power.
  
  Finally, insofar as the intelligence explosion is prevented or at least slowed down a lot, that gives more time for people, countries, and institutions that are currently asleep at the wheel (i.e. almost everyone) to realize the danger they are in and act to assert themselves before they lose their leverage. For example, workers can go on strike now + vote for regulations, but once AIs and robots can automate everything, workers will have less leverage.
  
  OK so that’s concentration of power. What about loss of control?
  
  Well, to prevent loss of control, it helps to (a) proceed cautiously with AI development, and *not* rush to put AIs in charge of important things (datacenters, factories, weapons) as fast as possible. (Reminder that the AI companies plan to do recursive self improvement, i.e. put AIs in charge of AI R&D within the company). It also helps to (b) have a better scientific understanding of how AIs work, how their goals/​values/​traits/​etc. are shaped by different kinds of training, how their ‘minds’ work so we can figure out what they are thinking, etc. (See: Chain of thought monitoring, mechanistic interpretability, j-space, etc.)
  
  Both of these things require going slower than max speed. They benefit from broadly deploying AI to many researchers and to the public, before putting AI in charge of dangerous things like AI R&D. They benefit from transparency, allowing the broader scientific community outside the AI companies to look at what’s going on and run their own experiments. (This itself is helpful for at least two reasons: One, it increases the brainpower devoted to the technical problems, and two, it decreases the bias/​conflict-of-interest inherent in having AI companies grade their own homework so to speak.)
  
  Finally, the transparency helps make the necessary regulations actually good (as opposed to e.g. incompetent regulations that don’t achieve their purpose and/​or cause lots of unnecessary damage) and helps verify that the regulations are in fact being followed (competitor companies and watchdog groups can look at what’s going on and call out stuff that seems like a violation of the rules, by contrast with a less transparent setup where the overworked regulator has to send in auditors or something to try to see if anything is amiss and then argue with the company about grey area cases).
  
  Anyhow, there’s a lot more to talk about obviously; there are downsides to total research transparency too which I haven’t discussed here. (See AI 2040 supplements for more). But this diagram explains the main ideas that make me excited about it.

• Daniel Kokotajlo 10 Jul 2026 20:19 UTC

  LW: 7 AF: 5

  2

  AF

  in reply to: Dan H’s comment on: AI 2040: Plan A

  Sorry that was an oversight, we’ll edit to include a footnote citing MAIM.

• Daniel Kokotajlo 10 Jul 2026 18:16 UTC

  6 points

  0

  in reply to: RHollerith’s comment on: AI 2040: Plan A

  That’s reasonable. We put Plan S in there as an available option because we do think it’s a serious proposal and might even be better than Plan A.
  
  

• Daniel Kokotajlo 10 Jul 2026 16:57 UTC

  16 points

  0

  in reply to: David Matolcsi’s comment on: AI 2040: Plan A

  If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.

  I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn’t really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It’s 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it’s basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they’ll probably have ASI within a few years of that...
  
  Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don’t think it’s obvious either way how it would go.

• Daniel Kokotajlo 10 Jul 2026 16:48 UTC

  16 points

  7

  in reply to: Forza’s comment on: AI 2040: Plan A

  I don’t currently think that BCIs will deliver a +3-4 SD boost. If I did, I probably would have written a different scenario. Can you say more about why you think they’ll have that big of an effect?

• Daniel Kokotajlo 10 Jul 2026 16:47 UTC

  8 points

  3

  in reply to: RHollerith’s comment on: AI 2040: Plan A

  It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
  
  I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it’s generally bad to restrict speech for the usual reasons. Lukas Finnveden’s comments elsethread are good. I’m open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.

• Daniel Kokotajlo 10 Jul 2026 16:27 UTC

  3 points

  1

  in reply to: dynomight’s comment on: AI 2040: Plan A

  Yep as we say in the piece, we think that it’s better for AI progress to happen mostly via scaling compute rather than mostly via algorithms (that is, in the context of a Plan-A style reversible deal with mutually assured chip destruction). So we think it’s a feature, not a bug, that publishing the algos disincentivizes investment in algo research. There’ll still be some algo progress in Plan A and that’s fine, we model it in the model. (Indeed, it’s probably impossible to solve alignment without making some algo progress as a side-effect)