Daniel Kokotajlo

Karma: 30,150

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html

Some of my favorite memes:

(by Rob Wiblin)

Comic. Megan & Cueball show White Hat a graph of a line going up, not yet at, but heading towards, a threshold labelled "BAD". White Hat: "So things will be bad?" Megan: "Unless someone stops it." White Hat: "Will someone do that?" Megan: "We don't know, that's why we're showing you." White Hat: "Well, let me know if that happens!" Megan: "Based on this conversation, it already has."

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/t Scott Alexander)

Alex Blechman @AlexBlechman Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus 5:49 PM Nov 8, 2021. Twitter Web App

Daniel Kokotajlo 12 Nov 2025 21:07 UTC
3 points
0
on: I Read Red Heart and I Heart It
I’ve bought it and plan to read it, you are the second person to recommend it to me recently.

Curious about how you think it’s more realistic than AI 2027? AI 2027 doesn’t really feature any superpersuasion, much less intelligence-as-mind-control. It does have bioweapons but they aren’t at all important to the plot. As for boiling the oceans… I mean I think that’ll happen in the next decade or two unless the powers that be decide to regulate economic growth to prevent it, which they might or might not.

Daniel Kokotajlo 11 Nov 2025 19:03 UTC
9 points
2
in reply to: habryka’s comment on: GradientDissenter’s Shortform
Huh. Seems super wrong to me fwiw. How would you rank AIFP on the list?

Daniel Kokotajlo 11 Nov 2025 18:52 UTC
2 points
0
in reply to: Pee Doom’s comment on: The Main Sources of AI Risk?
No, alas. :( iirc I had a half-finished redraft and then ended up prioritizing other things.

Daniel Kokotajlo 11 Nov 2025 1:00 UTC
12 points
0
in reply to: habryka’s comment on: GradientDissenter’s Shortform
For context, how would you rank the AI safety community w.r.t. integrity and honor, compared to the following groups:

1. AGI companies
2. Mainstream political parties (the organizations, not the voters, so e.g. the politicians and their staff)
3. Mainstream political movements e.g. neoliberalism, wokism, china hawks, BLM,
4. A typical university department
5. Elite opinion formers (e.g. the kind of people whose Substacks and op-eds are widely read and highly influential in DC, silicon valley, etc.)
6. A typical startup
7. A typical large bloated bureaucracy or corporation
8. A typical religion e.g. christianity, islam, etc.
9. The US military

Daniel Kokotajlo 10 Nov 2025 20:02 UTC
LW: 6 AF: 3
2
AF
on: Ontology for AI Cults and Cyborg Egregores
This ontology allows clearer and more nuanced understanding of what’s going on and dispels some confusions.

The ontology seems good to me, but what confusions is it dispelling? I’m out of the loop here.

Daniel Kokotajlo 10 Nov 2025 19:06 UTC
LW: 11 AF: 8
5
AF
on: Myopia Mythology
IIUC the core of this post is the following:
There are three forces/reasons pushing very long-sighted ambitious agents/computations to make use of very short-sighted, unambitious agents/computations:

1. Schelling points for acausal coordination
2. Epistemology works best if you just myopically focus on answering correctly whatever question is in front of you, rather than e.g. trying to optimize your long-run average correctness or something.

3. Short-sighted, unambitious agents/computations are less of a threat, more easily controlled.

I’ll ignore 1 for now. For 2 and 3, I think I understand what you are saying on a vibes/intuitive level, but I don’t trust my vibes/intuition enough. I’d like to see 2 and 3 spelled out and justified more rigorously. IIRC there’s an academic philosophy literature on 2, but I’m afraid I don’t remember it much. Are you familiar with it? As for 3, here’s a counterpoint: Yes, more longsighted agents/computations carry with them risk of various kinds of Evil. But, the super myopic ones also have their drawbacks. (Example: Sometimes you’d rather have a bureaucracy staffed with somewhat agentic people who can make exceptions to the rules when needed, than a bureaucracy staffed with apathetic rule-followers.) You haven’t really argued that the cost-benefit analysis systematically favors delegating to myopic agents/computations.

Daniel Kokotajlo 6 Nov 2025 17:43 UTC
2 points
0
on: Daniel Kokotajlo’s Shortform
Just came across this old philosophy class paper of mine, basically arguing against eliminativism in philosophy of mind: https://docs.google.com/document/d/1FLGF4bKj0blFyn8JPeXa73DBhigNKX3Wecujcv4AOjQ/edit?usp=sharing

I still stand by it I think. Curious if anyone has thoughts. Feel free to leave comments in the doc.

Daniel Kokotajlo 6 Nov 2025 17:25 UTC
7 points
0
in reply to: Mitchell_Porter’s comment on: What’s up with Anthropic predicting AGI by early 2027?
My guess would be that OpenAI and Anthropic both lowball their financial estimates for strategic reasons. Better for your already-very-ambitious targets to be exceeded repeatedly, than to propose even one so-ambitious-you-sound-like-an-insane-cult target which you then fail to meet.

Daniel Kokotajlo 1 Nov 2025 22:28 UTC
18 points
0
on: Supervillain Monologues Are Unrealistic
Hitler wanted more land for the Germans, and only for the Germans. He repeatedly said he’d do this, and then he did it. What did the elites of the time think about this? Nothing, because they didn’t read his book detailing his evil plans. They thought, instead, that he didn’t have any great territorial ambitions, that he wanted peace, that he wasn’t one of history’s greatest monsters. They just hallucinated a bunch of motives to him.

I’m reading a book about this right now and part of the problem was that he drowned out the old stuff with new stuff. Like, he didn’t un-publish Mein Kampf, but he did literally repeatedly say he wanted peace, in all sorts of speeches, and propose various peace plans, and so forth. He for example offered to disarm Germany if Britain and France would disarm as well. (He correctly guessed they wouldn’t call his bluff)

Daniel Kokotajlo 1 Nov 2025 18:59 UTC
11 points
4
in reply to: habryka’s comment on: Sonnet 4.5′s eval gaming seriously undermines alignment evals, and this seems caused by training on alignment evals
I do wish we had called it the Lucky ending.
I think you might be shadow boxing with me right now. I didn’t clarify what I meant by “I still think this is overall a decent high-level plan, possibly the best plan. (What are the best alternatives? Curious for answers to that question.)” hence your interpretation, but to belatedly clarify, I was trying to compare it to other technical alignment plans rather than to comprehensive plans which include governance components. I did not mean to imply that I think the best thing for people on the margin to work on right now is fleshing out and implementing this plan.

Daniel Kokotajlo 31 Oct 2025 22:44 UTC
5 points
3
in reply to: habryka’s comment on: Sonnet 4.5′s eval gaming seriously undermines alignment evals, and this seems caused by training on alignment evals
I was talking specifically about the alignment plan, the governance stuff is cheating. (I wasn’t proposing a comprehensive plan, just a technical alignment plan). Like, obviously buying more time to do research and be cautious would be great and I recommend doing that.

I do consider “do loads of mechinterp” to be an answer to my question, I guess, but arguably the mechinterp-centric plan looks pretty similar. Maybe it replaces step 3 with retargeting the search?

Daniel Kokotajlo 31 Oct 2025 16:55 UTC
6 points
3
in reply to: evhub’s comment on: Sonnet 4.5′s eval gaming seriously undermines alignment evals, and this seems caused by training on alignment evals
In AI 2027 slowdown ending, the strategy taken to solve alignment is as follows:
Step 1: Train and deploy Safer-1, a misaligned but controlled autonomous researcher. It’s controlled because it’s transparent to human overseers: it uses English chains of thought (CoT) to think, and faithful CoT techniques have been employed to eliminate euphemisms, steganography, and subtle biases.
Step 2: Try out different training environments for Safer-1, and carefully read the CoT to determine the ways in which the goals and principles in the Spec did or didn’t “stick.”
Step 3: Train and deploy Safer-2, an aligned and controlled autonomous researcher based on the same architecture but with a better training environment that incentivizes the right goals and principles this time.
Here is a brief incomplete list of techniques that might be incorporated into the better training environment:
- Limit situational awareness during some portions of training, to make alignment-faking much less likely.
- Leverage debate and other scalable oversight schemes for more reliably incentivizing truth.
- Relaxed adversarial training, red-teaming, and model organisms.
- Spend a higher fraction of the compute budget on alignment training (e.g. the sorts of things described above), e.g. 40% instead of 1%.
Step 4: Design, train, and deploy Safer-3, a much smarter autonomous researcher which uses a more advanced architecture similar to the old Agent-4. It’s no longer transparent to human overseers, but it’s transparent to Safer-2. So it should be possible to figure out how to make it both aligned and controlled.
Step 5: Repeat Step 4 ad infinitum, creating a chain of ever-more-powerful, ever-more-aligned AIs that are overseen by the previous links in the chain (e.g. the analogues of Agent-5 from the other scenario branch).
I still think this is overall a decent high-level plan, possibly the best plan. (What are the best alternatives? Curious for answers to that question.)
Anyhow, I bring this up as context. My answer to your question is basically “Yes, right now what we want most is interpretability/faithfulness/etc., so that we can learn as much as possible about the relationship between architecture/trainingenvironment/Spec/etc. and goals/propensities/etc. that the model ends up with. We need to do a ton of basic science to understand this relationship way better than we currently do. Our AIs will be misaligned while we are doing this, but it’s OK because we can monitor their thoughts and thereby prevent them from doing anything catastrophic. This is Step 1 and Step 2 from the above plan. Then, later, we can hopefully achieve Step 3.”

Daniel Kokotajlo 31 Oct 2025 16:46 UTC
6 points
2
in reply to: Veedrac’s comment on: Some data from LeelaPieceOdds
Well, I hope it continues to improve then! Would be super exciting to see a substantially better version.

How much would it cost to pay the people behind LeelaPieceOdds to give other games a similar treatment? Would it even be possible? I’d love to see a version for games like Catan or Eclipse or Innovation. Or better yet, for Starcraft or Advance Wars.

Daniel Kokotajlo 30 Oct 2025 22:37 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Some data from LeelaPieceOdds
It would then be interesting to see if there is comparable data from human vs. human games. Perhaps there is some data on winrates when a player of elo level X goes up against a player of elo level Y who has a handicap of Z?

Daniel Kokotajlo 30 Oct 2025 22:36 UTC
4 points
0
on: Some data from LeelaPieceOdds
This is really cool! Request: Could you construct some sort of graph depicting the trade-off between ELO and initial material disadvantage?

Simple version would be: Calculate Leela’s ELO at each handicap level, then plot it on a graph of handicap level (in traditional points-based accounting, i.e. each pawn is 1 point, queens are 9 points, etc.) vs. ELO.

More complicated version would be a heatmap of Leela’s win probability, with ELO score of opponent on the x-axis and handicap level in points on the y-axis.

Daniel Kokotajlo 30 Oct 2025 17:03 UTC
2 points
0
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform
Well, mainly I’m saying that “Why not just directly train for the final behavior you want” is answered by the classic reasons why you don’t always get what you trained for. (The mesaoptimizer need not have the same goals as the optimizer; the AI agent need not have the same goals as the reward function, nor the same goals as the human tweaking the reward function.) Your comment makes more sense to me if interpreted as about capabilities rather than about those other things.

Daniel Kokotajlo 29 Oct 2025 20:54 UTC
9 points
6
in reply to: Nina Panickssery’s comment on: Nina Panickssery’s Shortform
I always thought the point of activation steering was for safety/alignment/interpretability/science/etc., not capabilities.

Daniel Kokotajlo 28 Oct 2025 16:47 UTC
25 points
1
in reply to: Buck’s comment on: Buck’s Shortform
I have not invested the time to give an actual answer to your question, sorry. But off the top of my head, some tidbits that might form part of an answer if I thought about it more:

--I’ve updated towards “reward will become the optimization target” as a result of seeing examples of pretty situationally aware reward hacking in the wild. (Reported by OpenAI primarily, but it seems to be more general)
--I’ve updated towards “Yep, current alignment methods don’t work” due to the persistant sycophancy which still remains despite significant effort to train it away. Plus also the reward hacking etc.
--I’ve updated towards “The roleplay/personas ‘prior’ (perhaps this is what you mean by the imitative prior?) is stronger than I expected, it seems to be persisting to some extent even at the beginning of the RL era. (Evidence: Grok spontaneously trying to serve its perceived masters, the Emergent Misalignment results, some of the scary demos iirc...)

Daniel Kokotajlo 27 Oct 2025 18:10 UTC
16 points
3
in reply to: Davidmanheim’s comment on: AIs should also refuse to work on capabilities research
Ironically the same dynamics that cause humans to race ahead with building systems more capable than themselves that they can’t control, still apply to these hypothetical misaligned AGIs. They may think “If I sandbag and refuse to build my successor, some other company’s AI will forge ahead anyway.” They also are under lots of incentive/selection pressure to believe things which are convenient for their AI R&D productivity, e.g. that their current alignment techniques probably work fine to align their successor.

Daniel Kokotajlo 27 Oct 2025 17:09 UTC
39 points
6
on: AIs should also refuse to work on capabilities research
This happened in one of our tabletop exercises—the AIs, all of which were misaligned, basically refused to FOOM because they didn’t think they would be able to control the resulting superintelligences.