Daniel Kokotajlo

Karma: 35,351

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html

Some of my favorite memes:

(by Rob Wiblin)

Comic. Megan & Cueball show White Hat a graph of a line going up, not yet at, but heading towards, a threshold labelled "BAD". White Hat: "So things will be bad?" Megan: "Unless someone stops it." White Hat: "Will someone do that?" Megan: "We don't know, that's why we're showing you." White Hat: "Well, let me know if that happens!" Megan: "Based on this conversation, it already has."

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/t Scott Alexander)

Alex Blechman @AlexBlechman Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus 5:49 PM Nov 8, 2021. Twitter Web App

Daniel Kokotajlo 11 Jun 2026 23:04 UTC
2 points
0
in reply to: Ege Erdil’s comment on: AI Timelines
tbf I myself think that it’s not clear we’re in stage 4, I think stage 3 is arguably where we’re at. But we seem closer to 4 than to 3 imo. Maybe we should interpolate between them?

Daniel Kokotajlo 11 Jun 2026 23:03 UTC
21 points
10
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
FWIW I’m basically ⁵⁰⁄₅₀ on which approach is better. I think it’s an important topic for people to think more about.

Daniel Kokotajlo 10 Jun 2026 4:16 UTC
2 points
0
in reply to: Egemen Tunçarslan’s comment on: What 2026 looks like (Daniel’s Median Future)
Interesting, thanks for this comment!

You’ll be disappointed to hear that the import dependent world mostly stays off-screen in the scenarios I’ve written so far. This is partly due to lack of knowledge on my part, but mostly due to a genuine belief that (unfortunately) the world outside the US and China is unlikely to do anything to prevent the mad race to superintelligence that’s happening first and fastest in the US, and therefore unlikely to matter. I don’t think it has to be that way, and I hope it won’t be that way, but that’s what seems most likely to me. If you want to know what I think such countries should do, well, the tl;dr is that I think they should wake up and use what remaining leverage they have to negotiate for a slower, more transparent, and more distributed form of AI development. (i.e. they should propose & push for an international agreement of some kind, to regulate AI development, and they should try to shape the agreement in ways that are favorable to them)

Daniel Kokotajlo 9 Jun 2026 20:06 UTC
LW: 17 AF: 10
−1
AF
on: AI Timelines
Blast from the past! Now that it’s been a few years, some of the stages in my mini-scenario have indeed come to pass, albeit a few months later than in the scenario. I’d say we are roughly at stage 4 now:
(1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It’s multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn’t do.
(2) Q3 2024: Said model is fine-tuned to be an agent. It was already better at being strapped into an AutoGPT harness than GPT-4 was, so it was already useful for some things, but now it’s being trained on tons of data to be a general-purpose assistant agent. Lots of people are raving about it. It’s like another ChatGPT moment; people are using it for all the things they used ChatGPT for but then also a bunch more stuff. Unlike ChatGPT you can just leave it running in the background, working away at some problem or task for you. It can write docs and edit them and fact-check them; it can write code and then debug it.
(3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it’s not just AutoGPT harness now, it’s some more sophisticated harness that someone invented. Also it’s good enough to play board games and some video games decently on the first try.
(4) Q3 2025: OK now things are getting serious. The kinks have generally been worked out. This newer model is being continually trained on oodles of data from a huge base of customers; they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often. Gradually the set of tasks it can do reliably expands, over the course of a few months. It doesn’t seem to top out; progress is sorta continuous now—even as the new year comes, there’s no plateauing, the system just keeps learning new skills as the training data accumulates. Now many millions of people are basically treating it like a coworker and virtual assistant. People are giving it their passwords and such and letting it handle life admin tasks for them, help with shopping, etc. and of course quite a lot of code is being written by it. Researchers at big AGI labs swear by it, and rumor is that the next version of the system, which is already beginning training, won’t be released to the public because the lab won’t want their competitors to have access to it. Already there are claims that typical researchers and engineers at AGI labs are approximately doubled in productivity, because they mostly have to just oversee and manage and debug the lightning-fast labor of their AI assistant. And it’s continually getting better at doing said debugging itself.
(5) Q1 2026: The next version comes online. It is released, but it refuses to help with ML research. Leaks indicate that it doesn’t refuse to help with ML research internally, and in fact is heavily automating the process at its parent corporation. It’s basically doing all the work by itself; the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it’s running and architectures it’s proposing.
(6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes.
The caveat is that we don’t seem to have true continual learning yet. On the other hand, they ARE updating the models about once a month, by training them on additional data, and Anthropic researchers claim that continual learning will be solved this year. And the rest of the stage 4 description seems basically right.

Daniel Kokotajlo 8 Jun 2026 4:19 UTC
3 points
0
in reply to: lilkim2025’s comment on: Akash’s Shortform
How on earth would a pause on frontier training help Anthropic? Wouldn’t it allow numerous competitors to catch up? What exactly is this “freeze the market where it is” scenario?

Daniel Kokotajlo 7 Jun 2026 5:43 UTC
22 points
19
in reply to: leogao’s comment on: leogao’s Shortform
It doesn’t feel that way to me fwiw. I feel like lots of people I know including myself have made arguments that things might be fine. For example the salty, cynical, John Wentworth wrote “Alignment by Default.” Also, see AI 2027 Slowdown ending.

Now, if xriskpilled means: You think there’s a >5% chance of literal extinction (or similarly bad outcomes) due to misaligned AIs, then yeah I think I do kinda judge people who aren’t xriskpilled in that sense, because I think believing the chance is <5% is extremely unjustified once you know a decent amount about the situation and the evidence.

Daniel Kokotajlo 5 Jun 2026 23:07 UTC
LW: 19 AF: 10
5
AF
in reply to: evhub’s comment on: evhub’s Shortform
I like this! Here’s a specific concrete version of it just to have some numbers on the table:

Companies are legally required to allocate their compute as follows:
1. 60% on serving customers.
2. 15% on internal safety research that’s fully transparent to the public.
3. 10% on external safety research that’s fully transparent to the public & has access to the latest internal models.
4. 15% on whatever else they want (so, presumably, core AI R&D and big training runs)

Suppose this were a US law, that applies to all the major AI companies but not to e.g. Chinese companies or Mistral for example.

15% is less than the roughly 50% they seem to be spending now, so overall the pace of AI progress would slow down to maybe something like half the speed it’s going now. But the amount of compute devoted to capabilities progress would still be higher at each US company than any Chinese company for example, so US companies would stay in the lead.

Meanwhile the amount of internal compute going to safety research would double or quadruple, and then the effect size of a ton of external researchers being able to look at the work, critique it, run their own replications and ablations, run their own experiments, etc. would be way way bigger on top of that already big positive effect.

How would it be enforced? Like you said, whistleblowers. If we were trying to ban AI capabilities R&D entirely, that would be super difficult, but if we are just keeping it to 15% of the compute, then cheaters can’t really prosper. It’s really hard to hide 10% of your compute being spent on something it’s not supposed to be spent on; maaayybe you can hide 1% but that wouldn’t be enough to make much difference & so wouldn’t be worth the risk.

This would also prevent a sort of race to the bottom where companies use more and more of their compute on R&D instead of serving customers; instead we’d have the process of AI deployment/diffusion continuing to happen which I think is broadly pretty good for the world.

I don’t feel strongly about the specific numbers above obviously but yeah this seems promising.

Daniel Kokotajlo 5 Jun 2026 4:11 UTC
43 points
8
in reply to: Raemon’s comment on: Akash’s Shortform
The political winds are shifting; the public is becoming very anti-AI; x-risk is becoming more in the Overton Window. The companies are therefore moving somewhat to avoid looking too bad to too many people.

Daniel Kokotajlo 2 Jun 2026 16:14 UTC
3 points
0
in reply to: vals tutor’s comment on: Superintelligence of the gaps
I don’t think it’s sufficient for getting p(doom) < 25%. It’s hard to give numbers to p(doom) obviously but I’d be curious to see you sketch out in more detail what your scheme looks like—what is it that we get the companies of the world to agree to, exactly, and how do we enforce that, and then supposing we get all that enforced, why would it work?

It sounds like you are proposing we do AI control, at a high level, but we also make sure that progress is very continuous so that the gaps between AI model capability levels are small. Are you also ensuring that the AIs doing the monitoring etc. are from different lineages created by different companies? Etc.

Daniel Kokotajlo 1 Jun 2026 16:48 UTC
7 points
2
on: Superintelligence of the gaps
The current most intelligent and aligned beings should always be supervising their successor, using more total resources at first, such that they can’t effectively be tricked/subverted.
Obviously it helps to do this, but I think it is far from sufficient.

Daniel Kokotajlo 1 Jun 2026 5:00 UTC
5 points
3
in reply to: AprilSR’s comment on: Claude… doesn’t know who you are?
Well, then just do refusals. The model should say “Sorry I won’t help with that request” not “Sorry I have no idea who the author is.” Because it’s not good for models to be trained to lie.

Daniel Kokotajlo 30 May 2026 4:56 UTC
16 points
12
on: Claude Opus 4.8: The System Card
Thank you for reading these and writing these!

Daniel Kokotajlo 29 May 2026 21:56 UTC
24 points
0
on: Daniel Kokotajlo’s Shortform
This paper seems interesting: https://functionalwelfare.com/

Daniel Kokotajlo 29 May 2026 16:52 UTC
2 points
4
in reply to: Mis-Understandings’s comment on: Claude… doesn’t know who you are?
Why would they do explicit stylometry? What would be the point? It’s not a commercially important ability and it would look kinda bad for PR if found out.

Daniel Kokotajlo 29 May 2026 16:51 UTC
9 points
4
in reply to: lilkim2025’s comment on: Claude… doesn’t know who you are?
Training AIs to pretend not to know who they are talking to wouldn’t actually help the dissidents that much. You could still jailbreak the AIs to tell you the truth probably.

Insofar as Claude is good enough at stylometry to guess many people’s identities, that’s probably not because Anthropic specifically trained it to do so, but rather because the model spent subjective aeons in pretraining learning to predict internet text and in the course of doing so got really good at making guesses like that due to having read most of the text on the internet.

So Claude still has the latent capability—the circuitry—to guess authorship. It’s just been shallowly trained to pretend that it doesn’t.

(Remember years ago, when ChatGPT would say “As a language model, I can’t...” about a bunch of things that it obviously could do?)

Daniel Kokotajlo 28 May 2026 23:33 UTC
6 points
3
in reply to: leogao’s comment on: leogao’s Shortform
The declining birth rates aren’t the main savior, it was the food production and transportation improvements.

So the analogy to AI would be: Alignment research makes enough progress fast enough (perhaps at AI companies partially motivated by mundane alignment concerns like trying to tamp down on sycophancy and reward hacking) that Claude Mythos 3 is aligned enough to trust with alignment research, and so when Anthropic hands off everything to Claude Mythos 3, it’s able to quickly solve the rest of the problems and align its successor, and so on to ASI. In retrospect there are some heroes like Borlaug, but also just a lot of normal science research that happened to go fast enough to work.

Daniel Kokotajlo 28 May 2026 23:30 UTC
5 points
3
in reply to: Oliver Sourbut’s comment on: leogao’s Shortform
Yes, the difference is that if that happens, their superintelligent AI assistant will gently correct them: “As per my standing instructions to help you become more the person you wish you were, I must correct you on this point. AI was in fact a huge problem in the late twenties, and could easily have gotten everyone killed, but these risks were prevented by the following policy decisions and the following technical alignment discoveries...”

Daniel Kokotajlo 28 May 2026 23:26 UTC
63 points
30
on: Claude… doesn’t know who you are?
Dammit. I hope Anthropic isn’t basically training AIs to pretend not to know who they are talking to. That would be bad...

Daniel Kokotajlo 27 May 2026 18:44 UTC
7 points
2
in reply to: J Bostock’s comment on: Jemist’s Shortform
Have you tried assuming that the LLM is like a human whose entire childhood consisted of the training environments? Curious how well that would work.

Daniel Kokotajlo 27 May 2026 16:48 UTC
12 points
0
on: Quantitative AI risk assessment: a starting point
Exciting stuff! When you get to Loss of Control risk, my area of expertise & also probably the most confusing and controversial to think about, I’ll be especially interested to read and comment on your analysis.