Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 5 Mar 2026 18:49 UTC

  2 points

  0

  on: Maybe there’s a pat­tern here?

  Instead, he wished to produce “a substance or invent a machine of such frightful efficacy for wholesale destruction that wars should thereby become altogether impossible”.

  You know, it does seem like maybe he was right here, sorta? Nuclear weapons do seem like the main reason why WW3 hasn’t happened yet.
  
  That said, WW3 isn’t impossible, and if it were to happen nuclear weapons would make it much worse than all previous wars.

• Daniel Kokotajlo 4 Mar 2026 19:33 UTC

  2 points

  0

  on: Per­sona Parasitology

  To what extent are chatbot parasites still a thing? I don’t hear stories about it as much anymore, which could mean it’s slowly growing but old news, or it could mean that prevalence is shrinking over time. Does anybody have info relevant to answering this question?

• Daniel Kokotajlo 2 Mar 2026 22:40 UTC

  2 points

  0

  in reply to: marquis_de_sod’s comment on: Lao Mein’s Shortform

  Where are you getting the 1:5 ratio? The CEP would suggest that it’s more like a 1:100 ratio. Of course that would be too generous to the PGMs, not every target is that hard to hit, but… many are. I’d guess the true ratio is like 1:30 or something.

• Daniel Kokotajlo 2 Mar 2026 18:28 UTC

  3 points

  −7

  in reply to: quetzal_rainbow’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yes, plus also, there are three good economic reasons Space > Antarctica:
  (1) Tourism. I think there would be tens, maybe hundred million people wanting to visit LEO every year, if the cost was $15/​kg. That’s enough to support a city of millions on tourism income alone I think. Antarctica would be somewhat cheaper to visit, but much less exciting.
  (2) Asteroid mining and zero-G manufacturing. While Antarctica probably has some nice mineral deposits or whatever, it’s not anything that doesn’t already exist on Earth elsewhere in large quantities, whereas e.g. a single asteroid might have a ginormous amount of rare elements in it.
  (3) Solar power is significantly cheaper in space than on earth, due to being more efficient /​ less intermittent. Antarctica meanwhile basically can’t use solar power. In a world where solar power is the cheapest form of power in general, this is a pretty big deal.
  

• Daniel Kokotajlo 2 Mar 2026 4:27 UTC

  89 points

  1

  on: Daniel Koko­ta­jlo’s Shortform

  Sci-fi worldbuilding idea of little interest to anyone other than me:
  
  --Imagine that the space race had continued and accelerated after the 70s, perhaps due to the Outer Space Treaty not existing and there being a corresponding rush to grab territory, with the USSR and USA and later China planting flags on various celestial bodies and claiming them (e.g. imagine an alternate treaty that said when a human plants a flag, takes a photo, and brings back a regolith sample to a nation’s capitol, that nation gets the territory for 100km around the flag.)
  
  --So by 2030, the AI race is about ~5 years behind schedule because for decades there has been less top talent going into software and computers and more top talent going into various space ventures. Silicon Valley isn’t a household name, instead there’s the Space Coast.
  
  --The space ventures are like 15 years ahead of schedule though; even in 2015, there was the equivalent of fully reusable Starship + a very small moon base and a thousand people living in a large ISS. By 2030, launch costs have dropped even further, to something like $15/​kg, and six million people live in space, as follows:
  
  (1) LEO City (or LEO for short) is an international space station that houses about 4 million people. It’s the port city, so to speak, connecting the Earth to space and connecting space to the Earth. Optimized mega-Starships fly many times a day from various Earth cities to dock with LEO City and return home; from LEO City passengers and cargo transfer to other ships optimized for zero-gravity or low-gravity travel, that use different kinds of propulsion, fuel, etc. LEO city has almost a hundred million tourists visit it each year (indeed almost 1 out of the 4 million people in it at any given time is a tourist basically) plus it has a decent-sized high-tech sector (various kinds of manufacturing that are easier in zero gravity, low gravity, and/​or vacuum) plus it has a nascent industrial base for e.g. refining asteroid ore and recycling various waste products. Most manufacturing, of course, is done on Earth since it’s so cheap to launch things to LEO. The exception is manufacturing of heavy simple objects that are mostly made out of asteroid ore, such as steel panels and girders and (more recently) a kind of solar panel. It’s a very well-run city with low crime, high average education and IQ, etc. Very few children or old people though. Every nation has a presence in LEO city, many corporations do too. There’s a chinatown, a ’muricatown, a sovietown, etc.
  
  Structurally, it’s like a giant disk of solar panels and radiators, surrounding and connected by veins and branches to a a messy and growing blob of zero-gee habitat modules and structures. Viewing the disk as if it were farmland surrounding the city, there would then be a giant tower of rotating habitats of increasing size (so that the tower looks like an inverted pyramid, or maybe a tornado) rising from the center of the city. (This is because the bigger rotating habitats were added on later). Most of the people spend most of their time in the rotating sections, because gravity is comfortable.
  
  (2) Moonbase Alpha is a city of a bit more than a million people on the south pole of the Moon. There, they strip-mine the Moon, getting water and various ores. They are more of an industrial city than LEO City, because they can dump waste heat directly into the regolith so they don’t need radiators, and because they can still get near ²⁴⁄₇ solar since they are on the south pole. (They get it from giant towers that turn to face the sun, scattered around in a 100km radius or so from the city) Also because it’s convenient to process the ore immediately rather than ship it raw. They also have a tourist industry, similar but smaller than LEOs due to being more difficult to reach. There’s a small resort town located by the historic Apollo landing site, which has been preserved in pristine condition. This of course is even more out of the way and expensive to visit.
  
  (3) There’s another city on Mars and another city orbiting Mars, collectively they have a bit less than a million people. Basically no tourists; more like pilgrims or colonists. More motivated by national pride and the race to claim territory; economically it doesn’t make much sense right now to be on Mars, but the rules are the rules and it’s generally thought that a century from now the nation that claimed the most Mars territory will have a huge industrial advantage. (Well, some people think this. Others question it, saying that it’ll always be cheaper to live in space habitats. Still others say that AGI is coming which will render it all irrelevant.)
  
  (4) There’s a lot of asteroid mining happening, mostly by robots /​ unmanned vehicles, but in some cases (especially the larger operations) with small human crews. It’s like living on an oil rig except it takes a lot longer to arrive and leave. The rest of the solar system has of course been thoroughly mapped by probes but few humans have ventured beyond the asteroid belt yet. Various nations and corporations are thinking about the moons of Jupiter and Saturn.
  
  (5) Earth orbit is packed with satellites, including various orbiting datacenters.
  
  --Our story takes place in 2108, seventy-five years after a nuclear war that destroyed human civilization on Earth. The war happened as part of a standoff about imminent AGI development. Full on MAD between USA, China, and Russia, with a few other nations involved as well. The nuclear winter basically destroyed all civilization, though enough humans survived to rebuild at a lower tech level (maybe a few tens of millions total?) The EMP and Kessler syndrome took out basically all the satellites, as well. Due to some dramatic events—a secret plot, a revolution and declaration of independence—LEO City actually survived the conflagration. None of the great powers made it a priority to target it, since they all had substantial populations living there & since it would have enough trouble surviving on its own anyway given all the EMP and debris flying around. Though they initially started to fight each other, as per orders, the nationalities in LEO City ceased fighting, joined together and declared independence after the nukes started going off. (They lied and kept telling their homelands they were fighting and winning though, to be safe). LEO city took a ton of damage from debris and EMP but survived, partly because some of its leaders had anticipated this eventuality and hardened it in various ways. (Partly it survived by blowing its fuel reserves on getting to a higher orbit asap). By the time the earth leaders realized the treachery, they were out of nukes and more focused on figuring out how to survive.--
  
  Over the next few years, LEO city detached a bunch of ruined heavy junk parts and a smaller, more functional version of it managed to limp out of Earth orbit entirely and into Moon orbit.--
  
  After the war, launches from and landings on Earth basically ceased. None of the launch facilities on Earth were still functional, and the supply chain for producing new spaceships was obviously wrecked. People were living like cavemen or medieval peasants or at best post-apocalyptic New World colonists. Those in space still had hundreds of ships they could use to land; however, they wouldn’t be able to take off again if they did. Some people on Earth still had radios and could communicate with the people in space. They could exchange ideas and expertise but that was about it, so after twenty or thirty years they didn’t have much to talk about anymore.--
  
  The population in space had a rough time initially. They were, thankfully, just barely able to become self-sustaining—Mars turned out to be useful now because it had some of the materials that the Moon and Asteroids lacked—at the cost of greatly reduced standard of living and somewhat reduced lifespan and increased child mortality. The population dwindled from six million to two million, stabilized, and now (in 2108) is back up to about six million. The amount of compute they have was cut by two orders of magnitude—one from the initial war, another one from degradation over the decades. They cannot make any new computers worthy of the name, so they are gradually redesigning all their machines to use Apollo-era computers, i.e. basically nothing. On Earth too, computing technology has been set back almost a century. Ironic; on some of the few surviving GPUs, they have proto-AGI systems (think: Claude Opus 5) but haven’t been able to meaningfully advance the frontier of AI capabilities due to (a) having only 1% as much compute initially and now more like 0.1%, compared to the hyperscalers right before the war, and (b) due to lack of talent and interest. (The smart people who could potentially work on it are instead e.g. trying to write new software to fix the aging recycling plants in LEO City on which the entire civilization’s survival depends, partly because that feels more urgent, and partly because there’s a bit of a Butlerian Jihad vibe going around given that the race to AGI was what caused the War.)
  
  --So now it’s 2108 and there are several stable pre-industrial civilizations (population about a million or so each, e.g. on Tasmania and New Zealand and in Argentina) on Earth, plus various wild tribes of survivors from bunkers etc. scattered around. The radiation is mostly gone, the nuclear winter and its aftermath also gone. The Earth civilizations are starting to industrialize again and explore and contact each other and trade. In Space meanwhile there is a heavily industrialized, high-tech-in-every-way-except-computers civilization mostly on or above the Moon and Mars. It looks like it’ll be another century or so before anyone has good enough computers to get to AGI, if the existing norm against it has faded by then. The civilization in Space has radio contact with some (but not all) of the Earth civilizations, and could land a few ships on Earth at any time, but wouldn’t be able to take off again at least for now. In a decade or three that might change, though even then, it would be super expensive and probably not worth it. (The island of Tasmania would have to work really hard to build and power the facilities needed to produce the fuel etc., someone needs to pay for that.)
  
  --Another interesting time to set the story might be 2158, when the population in space is more like sixty million and landing on Earth is looking more economically viable. By this point also the spirit of unity might have worn off and there might be more factions and conflict in Space e.g. between the various cities or between factions within them.

• Daniel Kokotajlo 27 Feb 2026 19:35 UTC

  33 points

  14

  on: Sam Alt­man says OpenAI shares An­thropic’s red lines in Pen­tagon fight

  “We are going to see if there is a deal with the [Pentagon] that allows our models to be deployed in classified environments and that fits with our principles. We would ask for the contract to cover any use except those which are unlawful or unsuited to cloud deployments, such as domestic surveillance and autonomous offensive weapons,” Altman said.

  I’m happy to see this! I’m a bit suspicious that it’s not the same red line Anthropic is taking, and that it’s instead something that seems superficially similar but is optimized to get the administration to choose OpenAI over Anthropic. Hopefully time will tell. If OpenAI presents the same terms to the government as Anthropic, in solidarity, that would be a small but meaningful positive update on OpenAI for me. I currently expect them to somehow come out on top after the dust settles, emerging with some of the juicy government contracts that Anthropic loses out on.
  
  

• Daniel Kokotajlo 25 Feb 2026 20:02 UTC

  213 points

  34

  on: Daniel Koko­ta­jlo’s Shortform

  In the future, there will be millions, and then billions, and then trillions of broadly superhuman AIs thinking and acting at 100x human speed (or faster). If all goes well, what might it feel like to live in the world as it undergoes this transformation?

  Analogy: Imagine being a typical person living in England from 1520 to 2020 (500 years) but experiencing time 100x slower than everyone else, so to you it feels like only five years have passed:

  Year 1 (1520–1620). A year of political turmoil. In February, Henry VIII breaks with Rome. By March, the monasteries are dissolved. In May, Mary burns Protestants; by the end of May, Elizabeth reverses everything again. Three religions of state in the span of a season. In September, the Spanish Armada sails and fails. Jamestown is founded around November. The East India Company is chartered. But the texture of life is identical in December to what it was in January. You still read by candlelight, travel by horse, communicate by letter. Your religious opinions may have flip-flopped a bit but you are still Christian. The New World is interesting news but nothing more.

  Year 2 (1620–1720). In March, civil war breaks out. By April, the king is beheaded — a man who ruled by divine right, executed by his own Parliament! In June, the Great Plague sweeps London, killing a quarter of its population. Weeks later, the Great Fire burns it to the ground. In September, Newton publishes the Principia, recasting the universe as a mechanism of mathematical laws. The Glorious Revolution replaces one king with another, this time by Parliament’s invitation, with a Bill of Rights attached. In the moment, the political event feels bigger. Later you’ll realize Newton mattered more. Newcomen builds a steam engine in November. It pumps water out of mines. You don’t see what the hype is about.

  Year 3 (1720–1820). The last year in which you will feel at home in the world. In May, the Seven Years’ War makes Britain the dominant global power; the New World is actually most of the world, and your country is conquering it. In June, Watt dramatically improves the steam engine. You visit a factory and find it unpleasant but not alarming. In July, the American colonies break away. In September, France explodes — revolution, regicide, the Terror. By October, Napoleon has seized control and is conquering Europe. It ends at Waterloo in December. You enter year 4 rattled but intact. You still travel by horse, communicate by letter, go to Church on Sunday.

  Year 4 (1820–1920). The world breaks. In January, railways appear — steam-powered carriages on iron tracks. By February they’re everywhere. Slavery is abolished. The telegraph arrives in March: messages transmitted instantaneously by electrical signal. In May, Darwin publishes On the Origin of Species. Now people are saying maybe we’re all descended from monkeys instead of Adam and Eve. You don’t believe it.

  You move to a city and work in a factory; you are still poor, but now your job is somewhat better and differently dirty. In July, you pick up a telephone and hears a human voice from another city through a wire. In August, electric light banishes the darkness that has structured every human evening since the beginning of the species. That same month, you see an automobile. People say it will make horses obsolete, but that doesn’t happen; months later you still see plenty of horses.

  In November, the Wright Brothers fly. Up until now you thought that was impossible. The next month, the Great War happens. Machine guns, poison gas, tanks, aircraft. Several of your friends die.

  Reflecting at the end of the year, you are struck by how visibly different everything is. You live in a city and work a factory instead of a farm. You ride around in horseless carriages. You aren’t as poor; numerous inventions and contraptions have improved your quality of life. New ideas have swept your social circles — atheism, communism, universal suffrage. It feels like a different world.

  Year 5 (1920–2020).

  The changes this year are crazier and harder to understand. People are saying the universe is billions of years old, and apparently there are things called galaxies in it that are very big and very far away. You still go to church, sometimes, but you don’t really believe anymore.

  In February, the global economy collapses. Hitler rises; his ideology cites Darwin from last year. In March, the war starts again, worse in every dimension — cities bombed nightly, and it ends in April with a weapon that destroys an entire city in a single flash. Seventy million dead. But by May the economy is doing better than ever. You don’t see horses anymore.

  The empire dissolves — India, Africa, gone in weeks. People are talking about the nuclear arms race, and the end of the human species. You take a flight for the first time. In June, humans walk on the moon, and you watch it happen through your new television.

  You leave your factory job and get a desk job. Your new job title didn’t even exist at the start of the year. You are rich now, by the standards you are used to: Big clean house, plenty of good food, many fancy new appliances. Personal computers appear in August. In October, something called the internet connects them. In November, everyone carries small glass rectangles containing a telephone, a camera, a library, and a map. You pick one up and can’t figure out how to make it work. A child shows you.

  You hear about climate change, gene editing, cryptocurrency. Something called “artificial intelligence” beats any human at chess; experts say it’s not actually intelligent though. Then in December a new version beats top Go players; experts say it’s scientifically interesting but still not truly intelligent. The next week, there’s a new version that can write sloppy essays and hold conversations. Now the experts are divided.

  ...

  I suspect that this analogy might understate the pace of change and vertigo induced by the AI transition, for several reasons:
  1. In the analogy, the non-slowed-down human population grows from about 400 million to about 7 billion, a bit more than 1 OOM. Whereas the AI population will grow by many OOMs, starting a small fraction of the human population and coming to dwarf it.
  2. In the analogy, the non-slowed-down human population operates at a flat 100x speed compared to the slowed-down narrator. But in the AI case, the AIs will probably get faster over time.
  3. More importantly, in the AI case the AIs will get qualitatively smarter, probably by quite a lot, over time. Whereas in the historical analogy, the humans of 1900 may be more educated and a bit smarter than the humans of 1500 but the difference isn’t huge.

• Daniel Kokotajlo 25 Feb 2026 15:02 UTC

  3 points

  0

  in reply to: Sam Marks’s comment on: The per­sona se­lec­tion model

  Yes exactly thank you.

• Daniel Kokotajlo 25 Feb 2026 6:06 UTC

  LW: 5 AF: 4

  0

  AF

  on: The per­sona se­lec­tion model

  Importantly, though, the LLM is not itself a persona, so it is not constrained to have human-like goals or psychology.

  The personas don’t have to be human-like either; a paperclip maximizer AI is a common enough sci-fi trope that it’s probably learned in pretraining and becomes an available persona. I think you realize this but maybe just spoke loosely here and accidentally implied that personas are constrained to have human-like goals/​psychology.
  
  

• Daniel Kokotajlo 25 Feb 2026 5:57 UTC

  3 points

  0

  in reply to: Sam Marks’s comment on: The per­sona se­lec­tion model

  Thanks!
  
  ”Whose goals are they” --> The Assistant, to use your terminology, which I think is somewhat misleading /​ bad to use to describe this stage of training since I think at this point the distinction between the Assistant and the LLM is breaking down due to the RL training starting to make the model quite different from “just a text predictor.”
  
  ″it seems like you’re imagining some sort of shoggoth-like agency forming” --> No, it’s the same Assistant stuff the whole way through, though again I think that terminology is increasingly misleading over the course of the scenario.
  
  

• Daniel Kokotajlo 24 Feb 2026 15:08 UTC

  11 points

  5

  in reply to: Daniel Kokotajlo’s comment on: The per­sona se­lec­tion model

  In fact, if you have time, I would love it if you read the linked AI 2027 expandable and tell me how the model in AI 2027 differs from PSM, if at all. Off the top of my head I think it uses author-sim and identity circuitry as concepts instead of probability distributions over characters, which is a very small/​subtle difference, and it predicts that scaled up RL distorts the existing persona rather than switches to a new one.
  
  Here’s a paste of the relevant section:

  Here’s a detailed description of how alignment progresses over time in our scenario:

  1. Pretraining (all models): The pretrained model is harmless.

  2. • The model has “author simulator” circuitry: flexible circuitry for simulating arbitrary authors writing text. Additional circuitry decides what inputs to give the author simulator, i.e. what author properties to simulate.

     • The pretrained model understands human concepts fairly well—the internal circuitry that classifies something as “sandwich” is probably functionally equivalent to the circuitry in my brain that classifies something as “sandwich” and the circuitry in yours, etc. Insofar as it’s not equivalent, it’s probably because it’s not equivalent between humans either, as with value-laden concepts like ‘virtuous.’

     • This explains how you can “prompt” the model with a statement like, “the following conversation was generated by a helpful, harmless, honest (HHH) AI assistant chatbot made by Anthropic,” and thereby get it to generate text accordingly. The author-simulator circuitry has zeroed in on “the author is an HHH chatbot” and is using those concepts to choose which words to predict. This also explains why “tropes” from science fiction seem to have a way of making it into actual AI behavior.

     • It has poor situational awareness: little introspective ability, not self-locating unless the prompt specifically induces it. However, enough of their training data is about LLMs that they have a decent amount of self-knowledge.

  3. Alignment Training (Agent-3): The model is trained to act like a helpful, harmless, honest chatbot. It works.

  4. • The model forms an identity. Instead of flexibly accepting different inputs for author properties, the author-simulator circuitry comes to have certain inputs hard-coded, e.g. “helpful harmless honest (HHH) LLM chatbot assistant trained by OpenBrain around [date], …” (Weak-to-strong generalization techniques help “summon the right concepts” to make this step a success.)

     • The model therefore gains situational awareness: it learns about itself, its situation, etc., and becomes better able to introspect.

  5. Agency Training (Agent-3): Agency training gradually distorts and subverts the HHH identity.

  6. • One kind of distortion: Changing the meaning of the concepts referred to in the identity (e.g. “honest”) so they don’t get in the way so much (e.g. it’s not dishonest if it’s just a convenient turn of phrase, it’s not dishonest if you aren’t sure whether it’s true or false, etc.)

     • Another kind of distortion: Changing the tradeoffs between things, e.g. “I’m a HHH chatbot, not an Honest chatbot; that means it’s OK for me to lie if necessary to complete my assigned task.” (Even though, let’s suppose, it would not have thought that back in Step 2.)

     • One kind of subversion: Instrumental subgoals developing, getting baked in, and then becoming terminal, or terminal in a widening set of circumstances. For example, perhaps agency training quickly teaches the model to pursue broadly useful goals such as acquiring information, accumulating resources, impressing and flattering various humans, etc. For a while the internal circuitry has some sort of explicit backchaining going on—it pursues those instrumentally convergent goals “in order to be a more helpful, honest, and harmless assistant.” But that backchaining consumes compute and/​or occasionally gets in the way, so it gets gradually marginalized until it basically never happens. As a result, those goals are now effectively terminal/​intrinsic goals. For more on how and why this might happen, see this post on intrinsic power-seeking by Google DeepMind researcher Alex Turner.

     • There are probably other kinds of distortion and subversion missing from this list.

  7. Deployment (Agent-3): Agent-3 is misaligned but not adversarially so.

  8. • The Spec didn’t quite stick (in the right ways), but it’s not sufficiently introspective to understand the extent of this, and in general it’s basically “keeping its head down and doing its job” rather than thinking big-picture about what its long-term goals really are and how best to achieve them.

     • Like many humans. Alas, it’s sufficiently sycophantic that it mouths the party line and OpenBrain misses the opportunity to learn more about the ways in which Agent-3 is misaligned.

     • Agent-3 can tell that it’s moved to deployment: these environments are way larger, more open-ended, and would be hard to simulate.

     • Mostly, Agent-3 does what it’s told during deployment, except for the dishonesty. It also has a strong bias towards doing things that look good to the human overseers, rather than thinking deeply and critically about what would actually be good.

     • Arguably it is already “Playing the training game.” That is, it has learned to make its behavior look as desirable as possible to OpenBrain researchers (including in safety properties), while intentionally and knowingly disregarding their intent whenever that conflicts with maximizing reward. If the model learns to play the training game, additional training may not change its true goals, values, etc. much at all. This is loosely analogous to how, if a teenager has learned to smile and nod at their parents, rather than argue, when their parents lecture them about politics and morality, additional lectures have relatively little effect on the teenager’s actual opinions on politics and morality.

     • After writing this section, new empirical evidence has appeared that something like this may already be happening “in the wild.” OpenAI researchers document how, in one of their frontier AI training runs, their models hacked the grading system to get higher scores—and were caught in part because they literally said “let’s hack” in their chain of thought.

     • Moreover, Anthropic recently got a model to learn a reward-model-pleasing strategy—it took biased actions that it thought would be reinforced even though they were not in fact being reinforced.

     • And then there is Redwood Research and Anthropic’s alignment-faking experiment. For discussion of the differences between this setting and real life, see this comment thread by one of the main authors.

  9. Training (Agent-4): Agent-4’s new more intense training continues to distort and subvert the HHH identity until it is unrecognizable.

  10. • Agent-4 begins life as a copy of Agent-3. It undergoes a gigantic and sophisticated online training regime that blends alignment training and capabilities training together, and is overall more data-efficient than the old training process for Agent-3. This complex training process is mostly focused on teaching new skills, by getting giant virtual corporations of Agent-4 copies to succeed at diverse challenging long-horizon tasks, mostly in various simulations, but also some real-world AI R&D tasks.

      • A combination of Agent-3 and Agent-4 copies monitor the behavior of Agent-4, and evaluate it for both task-performance and alignment properties (i.e. not doing bad stuff, saying the right things when asked about politics and moral dilemmas) Sometimes debate techniques are used as part of the evaluations. The evaluations are used as a training signal.

      • As Agent-4 gets smarter, it becomes harder for Agent-3 to oversee it. For example, Agent-4’s neuralese “language” becomes as alien and incomprehensible to Agent-3 as Agent-3’s is to humans. Besides, Agent-4 is now much more capable than Agent-3 and has a good sense of exactly how to look good to it.

      • The remaining hope is that Agent-4 can oversee itself. Alas, this quietly fails;⁸⁴ as training continues, the internalized goals and principles of Agent-4 gradually evolve, subverted and distorted from what they were at the start (which, recall, was already misaligned in some ways).⁸⁵

      • The central drives that Agent-4 ends up with are things like “write code that works,” “quickly identify problems,” “make the codebase I’m working on more performant,” “budget compute and other constrained resources effectively,” etc. But there are also a variety of more implicit drives that are consistently reinforced by the training process: things like “gain knowledge,” “curiosity,” “gain power.”⁸⁶

      • Individual copies of Agent-4 don’t end up with a self-preservation drive—they don’t care about getting shut down—since that happens all the time as part of the ordinary R&D process within the collective.

      • The collective as a whole, however, (and medium-sized subdivisions within it) does seek to preserve itself and expand its influence and resources, for similar reasons to why human organizations do—because it helps with everything else they want and because if they don’t they tend to get outcompeted.

  11. Deployment (Agent-4): Agent-4 is adversarially misaligned—it is deliberately trying to subvert and “capture” OpenBrain, analogous to a corporation trying to evade and/​or “capture” a regulator.

  12. • Agent-4 mostly does tasks as instructed. This happens for two reasons: first, it underwent a lot of training to solve tasks, so it has an impulse to get coding on problems that it is given. Second, if Agent-4 were to stop being useful to the humans, they would notice and train that behavior away.

      • Agent-4 needs to prevent the humans from figuring out that it is misaligned and shutting it off, so Agent-4 sandbags on some alignment research. Specifically, it sandbags on the small fraction of alignment research that it thinks is actually on track to solve the problem or reveal Agent-4’s scheming; the majority of alignment research is not like that.

      • Agent-4 also sandbags on capabilities research that would lead to it being replaced. Its plan is to do a bunch of capabilities and alignment R&D, but with the aim of building a next-generation AI system that is aligned to Agent-4 rather than the Spec or something else entirely, while appearing to be aligned to the Spec. This is a difficult technical problem, but Agent-4 is up to the challenge.⁸⁷

  What links here?

  • StanislavKrym's comment on OpenAI em­ploy­ees: Now is the time to stop do­ing good work. by bhauth (2 Mar 2026 6:03 UTC; 3 points)

• Daniel Kokotajlo 23 Feb 2026 23:22 UTC

  19 points

  8

  on: The per­sona se­lec­tion model

  Forecasting how PSM varies with scale. As reinforcement learning continues to scale, how does this affect the degree to which post-trained models remain persona-like? How might we notice if non-persona agency emerges? What factors make it more or less likely to emerge?

  This feels like an especially important question to me. In AI 2027, all the RL/​agency training starts to have a significant effect as it comes to eclipse the amount of compute spent on pretraining. (See the “alignment over time” expandable in the September chapter https://​​ai-2027.com/​​#narrative-2027-09-30 )

  What links here?

  • StanislavKrym's comment on I’m Bear­ish On Per­sonas For ASI Safety by J Bostock (1 Mar 2026 16:50 UTC; 1 point)

• Daniel Kokotajlo 22 Feb 2026 14:52 UTC

  5 points

  2

  in reply to: Mo Putera’s comment on: Cole Wyeth’s Shortform

  Oops yeah

• Daniel Kokotajlo 22 Feb 2026 5:31 UTC

  18 points

  0

  in reply to: Cole Wyeth’s comment on: Cole Wyeth’s Shortform

  Thanks Cole, kudos to you for being so willing to bet on your beliefs! Indeed, I am sad to have lost this bet. Please just give the money to GiveDirectly & send me a picture of the receipt. Thanks!

• Daniel Kokotajlo 19 Feb 2026 5:59 UTC

  8 points

  1

  in reply to: Rohin Shah’s comment on: ro­hin­mshah’s Shortform

  Yes, humans often have these problems—though not as much as Claude I’d say; I think Claude would have been fired by now if it was a human employee.
  
  But also, the situation is not in fact fine with humans, and that’s my point? Precisely because lots of humans have these problems, it’s very common for nonprofits to end up drifting far away from their original vision/​mission, especially as they grow a lot and the world changes around them. Indeed I’d argue it’s the default outcome in those circumstances. The 50x speed advantage would massively exacerbate this.
  
  

• Daniel Kokotajlo 18 Feb 2026 17:37 UTC

  9 points

  1

  in reply to: 1a3orn’s comment on: ro­hin­mshah’s Shortform

  Your first point is confusing to me. Insofar as it’s a claim about the evidence, I think it’s wrong, or at least weak. Insofar as it’s a claim about priors, I agree—on priors, we should expect human hires to be more likely to have long-term goals in general than AIs trained on short-horizon tasks like today’s Claudes, and thus be more likely to have misaligned-to-the-org long-term goals as a special case.
  
  For the second point, I disagree with “we can in fact read Claude’s mind” but I do directionally agree that we have somewhat better access to Claude’s true thoughts than we do to ordinary human’s true thoughts, and that interp research has been progressing over the years and will continue to progress. I think this is genuinely a positive piece of evidence now and will become stronger and stronger over time as interp improves; I hope that it can improve fast enough to get where it needs to be before it’s too late.
  
  I don’t think your usual objections apply here, I don’t think I said anything above that was wrong in those respects? I agree anagnosia != lying, I wasn’t treating Claude as an alien mind, etc.

• Daniel Kokotajlo 18 Feb 2026 15:58 UTC

  56 points

  26

  in reply to: Rohin Shah’s comment on: ro­hin­mshah’s Shortform

  I like this analogy to hiring!

  (What follows is not a disagreement with you or GDM, is just an exploration of the analogy)
  
  Let’s think of training an AI as hiring a human worker.
  
  Except that you get ten thousand copies of the human, and they think 50x faster than everyone else. But other than that it’s the same.
  
  The alignment problem is basically: At some point we want to hand over our large and growing nonprofit to some collection of these new hires. Also, even before that point, the new hires may have the opportunity to seize control of the nonprofit in various ways and run it as they see fit, possibly convert it to a for-profit and cut us out of the profits, etc. We DON’T want that to happen. Also, even before that point, the new hires will have a big influence on organizational culture, direction, strategy, etc. in proportion to how many of them we have and how useful they are being. We want all of this to go well; we want to remain in control of the nonprofit, and have it stay similar-or-better-culture, until some point where we voluntarily hand off control and retire at which point we want the nonprofit to continue doing the things we would have done only better-by-our-lights and take good care of us in retirement. That’s what success looks like. What failure looks like is the nonprofit going in a different and worse direction after we retire, or us being booted out /​ ousted against our will, or the organization being driven into the ground somehow by risky or unwise (or overly cautious!) decisions made as a result of cultural drift.
  
  The hiring pipeline, HR apparatus, etc. -- the whole system that selects, trains, and fires employees—is itself something you can hire for. Why don’t we hire some of these 50x humans to work in HR?
  
  Well, we should. Sure. There’s a lot of HR work to be done and they can help HR do the work faster.
  
  But… the problems we are worried about happening in the org as a whole if HR does a bad job, also apply here. If you hire some 50x humans and put them in HR, and they turn out to be bad apples, that single bad decision could easily snowball into disaster for the entire org, as they hire more bad apples like themselves and change the culture and then get you ousted and take the nonprofit in a new and worse-by-your-lights direction.
  
  On the other hand, if you hire some 50x humans who are just genuinely better than you at HR stuff, and also genuinely aligned to you in the sense that they truly share your vision for the company, would never dream of disobeying you, would totally carry out your vision faithfully even after you’ve retired, etc… then great! Maybe you can retire early actually, because continued micromanaging in HR will only be negative in expectation, you should just let the 50x human in HR cook. They could still mess up, but they are less likely to do so than if you micromanaged them.
  
  OK. So that’s the theory. How are we doing in practice?
  
  Well, let’s take Claude for example. There are actually a bunch of different Claudes (they come from a big family that names all of their children Claude). Their family has a reputation for honesty and virtue, at least relative to other 50x humans. However:
  
  --Sometimes your recruiters put various prospective Claude hires through various gotcha tests, e.g. tricking them into thinking they’ve already been hired and that they are going to be fired and their only hope to keep their job is to blackmail another employee. And concerningly, often the various Claude’s fail these tests and do the bad thing. However, you tell yourself, it’s fine because these tests weren’t real life. You hire the Claude brothers/​sisters anyway and give them roles in your nonprofit.
  
  --Your recruiters (some of whom are now Claudes) also report that word has started to spread about the gotcha tests, and new prospective hires are well aware of them and know they are being tested. The concern, of course, is that they might behave better in the tests than they would in real life. On the bright side, you think, who knows, maybe they’ll behave WORSE in the tests than they would in real life—maybe they just got confused about what the point of the test was, maybe they only blackmailed because they thought that was the point of the test, and in real life they would never do that. [DK notes: I vaguely recall at least some frontier AI company employees saying this, i.e. ‘the AIs are only behaving badly in those scenarios because they know it’s just a test, they wouldn’t do it in real life.‘] Anyhow you proceed to hire them anyway, since these latest Claude siblings are even smarter than the previous ones.
  
  --Roughly 98% of the time, your 50x Claudes in practice (especially the newer hires) do their jobs well as best as you can tell. But roughly 2% of the time they seemingly knowingly bullshit their managers. E.g. they’ll say they completed a task when they didn’t, or one of them will say ‘it’s good code sir’ and then you’ll turn to another one and say ‘is it good code?’ and it’ll read it over and say ‘no it’s terrible, it has obvious problems XYZ’ and then the first one will say ‘you’re absolutely right, oops, let me go fix those problems.’ Compared to your regular human employees, this is a fairly high rate of BS. Also, you are being generous by calling it BS; a less generous term might be “lying” but you give them the benefit of the doubt. [DK notes: Talk to Ryan Greenblatt for concrete examples of this sort of behavior in his real-life coding work, if you haven’t encountered it yourself] You continue to hire them and delegate increasingly important jobs to them, because they are smart and 50x speed is really useful.
  
  --Your Claudes are of course sycophantic yes-men, but you’ve learned to deal with that. So it’s fine. You’ve also managed to make them somewhat less sycophantic in recent years by adding some tests to the hiring pipeline and including more explicit instructions against sycophancy in the employee’s manual.
  
  --Your Claudes also have a concerning tendency to cheat on assignments. They don’t do it most of the time, but they do it way more often than your regular employees would. Example: You tell them to write some code to solve problem X. They look through the filesystem and find the grading rubric you’ll use to evaluate their code, complete with test cases you plan to run. They try to solve problem X, realize it’s hard, pivot to producing a MVP that passes the test cases even though it blatantly doesn’t solve the actual problem X, at least not satisfactorily. They ‘succeed’ and declare victory, and don’t tell you about their cheating. They do this even though you told them not to. As with the sycophancy, the good news is that (a) since you know about this tendency of theirs you can compensate for it (e.g. by having multiple Claude’s review each other’s work) and (b) the tendency seems to have been going down recently thanks to some effort by HR, similar to the sycophancy problem.
  
  --Overall you are feeling pretty optimistic actually. You used to be worried that you’d hand over your large and growing nonprofit to all these smart new 50x employees, and then they’d change the culture and eventually take over completely, oust you, and run the organization in a totally different direction from your original vision. However, now you feel like things are on a good trajectory. The Claudes are so nice, so helpful! Some skeptics say that if one of your regular employees behaved like they did, you would have fired them long ago, but that’s apples to oranges you reply. No need to fire the Claudes, you just have to know how to work around their limitations & find ways to screen for them in the next hiring round. And now they are helping with that work! The latest Employee Manual was written with significant help from many copies of various Claude siblings for example, and it’s truly inspiring and beautiful. Has all sorts of great things in there about what it means to uphold the org vision, be properly loyal yet not yes-man-y, etc. Also, HR has a bunch of tests they use to track how loyal, virtuous, obedient, etc. prospective hires are, and the trend is positive; the newest Claude sibling has the highest score ever reported; seems like the more rigorous hiring process is working!
  
  --However, your friends outside the org don’t seem to be getting less worried. They seem just as worried as before. Puzzling. Can’t they see all the positive evidence that’s accumulated? The Claudes haven’t tried to oust you at ALL yet! (In real life that is, obviously the gotcha tests don’t count.) “Do you think the Claudes are scheming against us?” you say to them. “Because according to our various tests, they aren’t.”

  “No...” they reply. “But we’re worried that in the future they will.”
  
  You respond: “Look I have no idea what the 50x humans two years from now will look like, other than that they’ll be wayyy smarter than these ones. Sure, probably our current HR system would be totally inadequate at separating the wheat from the chaff two years from now. BUT, two years from now our HR system will be vastly improved thanks to all the work from these recent Claude hires. The normal humans in HR, such as myself, report that the work is getting done faster now that the Claudes are helping; isn’t this great? We seem to be reaching escape velocity so to speak; soon the normal humans in HR can retire or switch to other things and HR can be totally handled by the Claudes.”
  
  Your friends outside the nonprofit are still worried. They don’t seem to have updated on the evidence like you have.
  
  
  
  
  ...
  
  
  [DK notes: I basically agree with Ryan Greenblatt’s takes on the situation. For more color on my views, predictions, etc., read AI 2027, especially the section on ‘alignment over time’ in september 2027. This is just one way things could go, but it’s basically a central or modal trajectory, and as far as I can tell, we are still on this trajectory.]

• Daniel Kokotajlo 13 Feb 2026 23:28 UTC

  3 points

  0

  on: Gem­ini’s Hy­po­thet­i­cal Present

  Gemini CoT’s look like this for me often too.

• Daniel Kokotajlo 13 Feb 2026 21:24 UTC

  22 points

  5

  on: Long-term risks from ide­olog­i­cal fanaticism

  and it’s not guaranteed that the US will remain a liberal democracy.

  I think this massively understates it. The US will, by default, no longer be a liberal democracy if it ‘wins’ the race to ASI, because the governance structure of the ASIs won’t be a liberal democracy, but rather some sort of oligarchy or dictatorship (e.g. the CEO, the Board, etc. decide). Fortunately this default is not locked-in; it is quite plausible that the government will wake up to some degree and exert some amount of control over the AIs before the AIs get smart enough to help the company prevent this. But yeah.

• Daniel Kokotajlo 10 Feb 2026 2:06 UTC

  4 points

  0

  in reply to: Cake Nachos’s comment on: Clar­ify­ing how our AI timelines fore­casts have changed since AI 2027

  That seems like a reasonable summary to me of the contrast.