paulfchristiano

• paulfchristiano 12 Sep 2026 6:00 UTC

  15 points

  0

  in reply to: charding’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  I did not have any OpenAI equity at the time I joined NIST and do not have any now. I signed the standard agreement you referenced, though I don’t agree with that post’s legal or practical analysis.

• paulfchristiano 11 Sep 2026 7:59 UTC

  133 points

  8

  in reply to: orthonormal’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  I haven’t addressed it publicly. Here’s a timeline and some root cause analysis:

  • When I left OpenAI I negotiated to make sure I could use everything I had learned about safety in my future work. This was a supplement to the normal severance agreement. I had a lawyer review the negotiated supplement but neglected to have them review the main agreement, which I think is the most important root cause. It’s also just a kind of extreme mistake and I’m not sure how I made it.

  • I did review the standard severance agreement myself, and looked for anything that would be problematic. It contained a prohibition on making “disparaging, derogatory, defamatory, libelous, or scandalous statements,” which I failed to flag as problematic. I think that was a mistake.

  • I vaguely remember or can reconstruct some of my reasoning here though it was a long time ago. For the purpose of understanding how I made the mistake, here are some things that I think were going on:

    • I compared the agreement to another severance agreement (from Google) which had similar terms. I mistakenly thought that signing severance agreements was much more common than it actually is. This is related to OpenAI being my first job out of my PhD.

    • Because it was an agreement about leaving the company I was thinking about this clause in the context of making negative statements about the circumstances of my departure. I was leaving on good terms and didn’t have any desire to make negative statements about my departure.

    • The only non-disparagement agreement I’d been asked to sign before this (in a non-OpenAI context) was more stringent: it prohibited any statement that was “reasonably likely” to degrade the counterparty’s reputation. I refused to sign that agreement, and that’s what I had in mind as a prototypical “non-disparagement agreement.” (Incidentally I also didn’t consult a lawyer in that case, suggesting that younger-Paul had a pattern of very bad legal judgment.)

    • If the line had just said “disparaging” instead of “disparaging, derogatory, defamatory, libelous, or scandalous” statements, I probably would have flagged it. Having many other extreme items in the list makes it seem like “disparaging” also means something extreme.

    • In normal language “disparaging or derogatory” would often imply a statement was actively insulting and not merely negative. That would still be bad to sign but much less bad. I very rarely disparage anyone in this narrower sense.

    • The offending text is two words in the middle of paragraph 14, which turn out to be more important than the whole rest of the document. That makes it easier to spend correspondingly less time double checking an initial impression. (Just another good reason to have a lawyer.)

    • I was leaving on good terms with OpenAI and didn’t expect OpenAI to play hardball on this agreement (except for confidentiality issues). I felt like escalating any of this stuff would look really bad for them and I was pretty happy to defend myself rather than cave (partly because I had plenty of resources). I think this isn’t a good reason to sign a bad agreement but it’s relevant to understanding why I wasn’t more stressed about the situation.

  • After signing the agreement I did not think of myself as having any ongoing restrictions on what I said about OpenAI, and I forgot about this clause altogether pretty quickly (for basically the same reasons that I didn’t initially flag it as problematic).

  • When I later heard complaints about non-disparagement agreements I did not connect them to what I had signed. Around the time the Vox article came out I checked over my agreements and realized that I had in fact signed the same thing. This was shortly after I had started at AISI. Over the next week:

    • I raised this issue with a supervisor (with a JD) who was dismissive of the possibility that this would interfere with my responsibilities in any way in practice.

    • I wrote up a document and shared it privately with some rationalists.

    • I discussed publication with AISI and was lightly discouraged from publishing. In general I wrote very little on any topic at the start of my time in government because I was encouraged to be pretty conservative as a high-profile civil servant.

    • If I had thought the issue was important I would have published it anyway, but I didn’t think it was a priority. I felt pretty confident it hadn’t changed any of my speech since I had quickly forgotten about it. A week later the agreement was voided, and then I dropped it altogether.

  Overall I consider this a failure of prudence (especially the failure to consult a lawyer) rather than integrity.

  What links here?

  • orthonormal's comment on Per­sonal state­ment on join­ing the OpenAI non­profit board by paulfchristiano (9 Sep 2026 18:58 UTC; 17 points)
  • Erich_Grunewald's comment on Per­sonal state­ment on join­ing the OpenAI non­profit board by paulfchristiano (12 Sep 2026 12:05 UTC; 2 points)

• paulfchristiano 9 Sep 2026 19:29 UTC

  88 points

  1

  on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  Some notes on public communication, which I expect the LW audience is especially interested in:

  • I believe the only legal restrictions on my speech in connection with OpenAI are non-disclosure agreements with OpenAI PBC and my fiduciary duty to the OpenAI Foundation (separately I also have confidentiality obligations to the US government and confidentiality obligations at ARC). The main thing to have in mind when interpreting my public comments is that you shouldn’t expect me to say anything based on confidential information.

  • A fiduciary duty implies a duty of confidentiality, including confidentiality about the deliberations and process of the board. I think the implications for speech beyond that are mild, but I think it does mean e.g. that it would be inappropriate for me to try to litigate a boardroom conflict via public statements. I have no legal concerns about publicly stating my own beliefs based on public information even when they are about OpenAI or what OpenAI should do.

  • I believe I will inevitably be interpreted by some audiences as speaking for the OpenAI nonprofit or speaking with particular authority about OpenAI, even when speaking in a personal capacity. I think it’s appropriate for me to be mindful of that, and it is unlikely for someone to be in this role if the board did not expect them to be mindful of that. So although I don’t have legal restrictions, I think you should expect my public communication to be more reserved than if I were not on the board, especially when it comes to comments about specific AI companies. (In this post in particular I tried to avoid making any implied comparisons between different AI companies for this reason.)

  • I made the tone in this post softer than I would have if I were not a director, although I think it accurately and clearly reflects my beliefs. In general I think it is appropriate for me to get feedback from the OpenAI nonprofit when communicating on particularly sensitive topics and that should be your default expectation for any public statements I make directly related to OpenAI. I may also get feedback from the for-profit but consider it more advisory.

Per­sonal state­ment on join­ing the OpenAI non­profit board

• paulfchristiano 18 Aug 2026 4:52 UTC

  3 points

  0

  in reply to: Raye’s comment on: Re­turn­ing to ARC

  Yes, still open here!

• paulfchristiano 9 Aug 2026 2:49 UTC

  2 points

  0

  in reply to: sscg13’s comment on: Re­turn­ing to ARC

  From ARC’s perspective we want to explain the interesting properties of NNUE (for us that’s exactly what it means to “interpret” a model). By far the most interesting behavior of NNUE is that it plays good chess, though there are also various closely related properties like “Its output is highly correlated with the winner of the game.”

  For any such properties you would need to start by ensuring that you can mechanistically analyze chess games appropriately. E.g. for a given position under random play you should have a mechanistic estimate for the the probability that white wins which is more accurate than sampling. I think that’s a really interesting and probably very tractable problem—I expect a solution to involve tracking how various features evolve over the course of the game, and then to track increasingly high-order interactions between features as you improve your precision.

  I think this is roughly what you mean by “the behavior of search alone is still extremely difficult to understand, despite being mechanistically open,” although perhaps I’m saying something slightly more extreme in that I’m saying even the outcome of random play is nontrivial to understand. There isn’t a neural network for which we need an explanation, and indeed we think that the algorithm for mechanistically predicting the outcome of chess games should be very simple (roughly speaking we think it should be no more complex than the rules of chess themselves).

  If you did that then it would be very reasonable to look at NNUE and estimate e.g. why its outputs are correlated with the outcome of the game under random play. A good understanding of NNUE should let you do that (or even estimate the gradient of that correlation with respect to the weights). I think that’s a very ambitious project that would be a huge deal for ARC if it could succeed, I’d say maybe ~50% of the way to explaining an LLM.

• paulfchristiano 7 Aug 2026 17:52 UTC

  7 points

  2

  in reply to: another-anon-do-gooder’s comment on: Re­turn­ing to ARC

  It’s also entirely possible to avoid an AI takeover in the 20-30% of worlds where existing methods can’t scale to superhuman AI (or in the worlds where they do scale but our implementation falls short). In particular we might develop new methods that do solve the problem or coordinate not to build uncontrollable AI.

  I was clarifying that I don’t mean “this is just a cakewalk in 70-80% of worlds.” There’s enough failure probability in those worlds that I think the default thing for a technical person to do is try to reduce it.

  I generally think we’re going to need to rise to the occasion one way or the other.

• paulfchristiano 7 Aug 2026 15:32 UTC

  12 points

  2

  in reply to: Roko’s comment on: Re­turn­ing to ARC

  It might well fail, but it isn’t really related to P=NP.

• paulfchristiano 7 Aug 2026 5:21 UTC

  5 points

  1

  in reply to: Christopher King’s comment on: Re­turn­ing to ARC

  You would never learn “The model is not trying to play chess well” (the word “trying” will never appear in any outputs from this system). But when you ask “How likely is the model to win this game of chess?”, you’ll get a lower number in games where the model is not trying (because the explanation necessarily captures the mechanisms by which “trying” causes the probability of victory to be higher).

  Similarly for takeover attempts, you won’t learn “the model is trying to take over.” But if you ask your estimator “What’s the chance that something would look horribly wrong if I investigated this incident?” the probability will spike when the model starts trying to take over.

  (Unless it takes over in such a way that you would not notice even if you investigated. To get around that you could try to say “Well what if I shut down all the AI and did a big investigation at the moment when it would be most damning?” I think that getting answers to this kind of question would be a huge deal and would get us most of what we want /​ would require solving the core difficulties in alignment. but it’s still far from bulletproof and at some point we need to roll up our sleeves and deal with the philosophical work I mentioned about referring to the causes for patterns in our observations and not just the observations themselves.)

• paulfchristiano 6 Aug 2026 15:06 UTC

  7 points

  0

  in reply to: Azimov’s comment on: Re­turn­ing to ARC

  Right now we are working on mechanistic explanations for computational phenomena, for which the “meaning” is defined precisely. In phase 2 of the project we’ll extend that to observable phenomena for which definitions are still straightforward. The main philosophically tangled part is pointing to latent structure for which no good definition is available, for which we want to say something like “The presence of a diamond is the property that causes such-and-such observable regularities,” and where I have some thoughts about why that should work but it’s far from a complete story.

  The main difference between a mechanistic loss function vs a behavioral loss function is that an adversarial model can control their behavior to make it look the same as the desired model on the training distribution, such that a behavioral loss function can’t distinguish the two. Whereas a mechanistic loss function detects how a model is thinking and if that would lead to undesirable generalization then it can get directly penalized. Or put differently a behavioral loss function cannot depend at all on how the model behaves out of distribution, and so ends up relying on an empirical bet about generalization, whereas a mechanistic estimate can try to address that behavior. In analyzing such an algorithm we’d still evaluate it against cases like “an adversarial model is trying to do get a low loss” and confirm that this results in good behavior, or even “a schemer is trying to take over and might want to get a low loss instrumentally” and confirm that our methods can detect and penalize that bad behavior OOD.

• paulfchristiano 6 Aug 2026 15:01 UTC

  8 points

  1

  in reply to: Christopher King’s comment on: Re­turn­ing to ARC

  ARC does expect to be able to “explain” how a chess-playing AI works, though we expect that to be pretty challenging. It avoids some of the issues with an unknown input distribution, but in terms of the mechanistic explanation itself I think it’s going to be just as hard as an LLM.

  We think such an explanation should:

  • Let you predict how often a chess AI wins against another AI from a given position using a mechanistic analysis of the rules of chess and the dynamics of the network, with strictly better computational efficiency than just playing a bunch of games. (Note that predicting who wins between two AIs requires understanding both of those models, for neural networks those explanations will be learned by gradient descent, but for a hand-coded chess algorithm we’ll need to do some hand-coding of an explanation that we conjecture to be not much harder than the design of the original algorithm.)

  • It should let you answer a range of questions about the AI that should intuitively be easy “if you understood how the model works.” That would include finding certain kinds of adversarial examples, by identifying and then violating assumptions that the model’s performance is predicated on. Perhaps more importantly, if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens and recognize changes that would reduce that probability.

  • By the same token it might let you extract the rules of chess to the extent that you had a precise question to which “the rules of chess” were the answer, but would definitely not let you explain the rules of chess to a human (since no humans were involved in the definition of the process). And if you had the rules of chess you would then be able to beat sampling in estimating how often the model would try to make an illegal move and would be in a better position to find cases in which it tried to make an illegal move (though again this might require some chess-specific labor, whose output would be simpler than the rules of chess).

• paulfchristiano 5 Aug 2026 14:49 UTC

  LW: 14 AF: 7

  4

  AF

  in reply to: habryka’s comment on: Re­turn­ing to ARC

  By “existing techniques” I do include “fiddle with the RL environments” or “midtrain on some documents about the intended behavior” or “run a prompted monitor over traffic in prod” or etc.

  When I say “understand and shape generalization” I think the central example is (i) adjust parameters of the training process that don’t affect competitiveness, (ii) build some understanding of how those parameters affect generalization so that you can adjust them in a helpful way. I think you’re saying that’s not a “technique;” I don’t care much about the semantics. I’m sure I put a higher probability on those changes helping than you do but it doesn’t seem worth arguing about here.

• paulfchristiano 5 Aug 2026 4:22 UTC

  LW: 49 AF: 18

  13

  AF

  in reply to: habryka’s comment on: Re­turn­ing to ARC

  By saying these techniques “break down” I mean “they cannot be used to get competitive work out of an AI system without having it take over.” I believe that:

  1. Existing AI systems won’t take over. I think there are still a few lines of defense before existing AI systems pose a significant risk of takeover (though it is not clear how long these will last).

  2. Modern AI systems are not pushing the limits of existing methods. I think you could push those methods much harder in order to significantly reduce the probability of takeover.

  In my mind that latter point is one of the main arguments against working on a project like ARC. I think it’s fairly likely that we live in one of the 70-80% of worlds where existing methods can in principle scale to broadly superhuman AI, but that we still get an AI takeover because our implementation isn’t good enough.

• paulfchristiano 5 Aug 2026 1:23 UTC

  16 points

  1

  in reply to: David Johnston’s comment on: Re­turn­ing to ARC

  Your reading makes sense and I don’t want to quibble over semantics (or foreclose potentially productive dialog). I do think there’s an important distinction between:

  1. Optimize a loss function that captures performance on the training distribution, and then shape the architecture /​ curriculum /​ regularizer /​ etc. so that the model behaves in a desirable way out of distribution.

  2. Define a loss function that captures performance off of the training distribution (e.g. what would happen in conditionals that we can’t sample in the lab) and optimize that.

  I was describing ARC as trying to pursue option #2.

  That said, in reality we are going to need to characterize the real world using learned hypotheses, since we can’t give a formal characterization without using ML. And then of course we care about how those hypotheses generalize. So regardless of the distinction above I do think it’s fair to characterize our work as an example of “understanding and shaping ML generalization.” We’re just willing to make much larger changes to get the generalization we want. In particular we are not trying to understand or shape how an agent generalizes, but instead are trying to separate the learning of hypotheses about the data (which do need to generalize well) from mechanistic analyses of predictors and agents that behave well under those hypotheses.

  What I really care about is that our methods would likely work even if competitive RL converges robustly to training-gaming behavior, while I’m claiming that other approaches rest on the empirical bet that what generalization you get is sensitive to some training parameters that we can tweak without big performance hits.

• paulfchristiano 5 Aug 2026 1:06 UTC

  20 points

  2

  on: Ob­sta­cles in ARC’s agenda: Find­ing explanations

  Coming back to this (now that I’m back at ARC) I think we have much clearer answers to most of these questions in 2026 than we did in 2025. I don’t know if these answers will necessarily be more comforting to you but they are at least much bolder: less ambiguous, more falsifiable. I think that’s good news for us either succeeding or failing faster.

  once we acknowledge that there are phenomena within a neural net that can’t and shouldn’t be explained, it’s unclear what our exact goal is and where we draw the line, what can be classified as an empirical regularity and what needs to be further explained.

  At the time David wrote this we were imagining that some logical facts (like “a queen is worth 9 pawns”) would get treated as inexplicable empirical regularities. That was scary, because if you treat the value of a queen as a brute empirical fact then why don’t you treat “the model tells the truth” as a brute empirical fact?

  We no longer believe that. Now we think that in a purely computational system we explain everything. We only need to deal with empirical regularities when we have data drawn from an unknown input distribution. So e.g. we should be able to mechanistically predict that a model is 96.2% less likely to win a chess game if it loses its queen (given that its playing against a particular opponent), with accuracy strictly greater than simply playing out a bunch of games to check.

  There is still real work in distinguishing “learning about the data distribution” from “testing computational machinery.” (I’ll return to this at the bottom of the comment.) But the conceptual line between them is now pretty clear and explaining purely computational systems is now a clean subproblem. In particular, if we fail at it we won’t be able to throw up our hands and say “it’s all an empirical regularity.”

  In ARC’s current thinking, the plan is not to have a clear line of separation between things we need to explain and things we don’t. Instead, the loss function of explanation quality should capture how important various things are to explain, and the explanation-finding algorithm is given a certain compute budget to build up the explanation of the model behavior by bits and pieces. [...] my main position is just that solving capacity allocation is crucial for explaining anything interesting at all, and it seems extremely hard, and we don’t know how to do it.

  David was going a bit out on a limb here since ARC didn’t really have a clear view in 2025. But fortunately in the last year the plan has gotten much more concrete, and it looks like we don’t need anything like capacity allocation.

  Back then we imagined an estimator G(X|pi) that estimates a quantity X given an argument pi.

  Now we imagine a family of estimators G_m(X|pi) parametrized by the computational budget m. If the quantity X is chosen randomly then pi can be empty and the estimates G_m(X) should be accurate (that’s what happens when we evaluate random neural networks). If instead X is highly structured then pi is suppsoed to “explain” that structure. If we have a structured neural network with a trillion parameters, we are hoping for a single explanation pi with roughly a trillion parameters that lets us answer random questions about that neural network, in the same way that the unconditional G_m(X) is good for random quantities X.

  (That’s a tiny bit too crude—we want a pi that lets us answer typical questions, but if we pick out a very specific question of n bits then we may need to supply another n bits of advice to complement the trillion parameters of pi. You can see the rough formalization for this here.)

  The upshot is that we aren’t picking pi to explain specific questions, we are learning it over the course of training to make sense of the model itself. We acknowledge that there is an infinite list of unimportant unstructured details, but those are now absorbed into the work done as G_m(X) spends more time thinking about X, rather than into pi itself. There are still conceptual issues here as we combine this with empirical regularities, but the situation is much cleaner than it was last year.

  We can’t really say we are fully explaining why the AI has high accuracy on a set of tasks, as the explanation would need to involve facts about the outside tools (like the internet) the AI is using. If we are only allowed to refer to the things the AI believes about the world, is our explanation about why the AI believes it will have a high accuracy?

  Our goal is explaining why the AI has high accuracy on a set of tasks. The explanation needs to involve facts about the environment, including e.g. “When I type a mathematical expression into the calculator, I get an accurate result.” We are going to represent those facts about the environment using the same kinds of flexible representations that we are using in ML. The loss we use for those facts will be a prediction loss, though rather than predicting bits we will be predicting various intermediate quantities.

  Here we do face a capacity allocation question, not in training explanations but in deciding what facts about the data distribution to model. We’d like to model roughly the same facts the AI does. The AI “learns” things by gradient descent minimizing the loss, and we think that if we learned everything about the environment that is needed to predict the loss then that will capture (at least) everything that the model could have learned about the environment.

  This area hasn’t gotten as much love as the prior two questions, but I feel pretty good about it overall. I think if we’re going to fail, we’ll probably also fail to explain how and why neural networks work in well-defined computational environments (like a Python interpreter).

  Similarly, if the vault-protecter AI is allowed to design a new AI, or just hire humans to do some tasks, it is very dangerous to rely on teleological explanations of the new AI or humans following the goals given to them, as you still can’t distinguish if they are achieving the goal through legitimate means.

  I agree! If your AI builds another AI to do the job, then you will be uncertain about whether the new AI is aligned and how it will generalize. Your AI will also be uncertain about whether the new AI is aligned.

  ARC’s position on this is that there’s no guarantee that an aligned AI will solve alignment of the further AI systems it builds (just like there’s no guarantee that it will successfully handle any of the other challenges it faces). ARC will try to help the automated AI researchers in exactly the same way we try to help the human AI researchers (i.e. by coming up with great alignment techniques it can use), and AI companies will try to help the automated AI researchers by making them smarter and wiser. But this is not the kind of agenda that gives you a promise that all future AI systems will be aligned, it’s the kind of agenda that tells you how to align AI systems in the current paradigm.

  ---

  I want to talk about empirical regularities one more time since it’s a recurring theme in this post:

  This means we will need to draw the line somewhere between things that are obviously unexplainable (‘Barack’ followed by ‘Obama’), and things that are complex enough that catastrophic deceptive alignment can hide inside the observed phenomenon (the model always behaves nicely when we ask it to).

  To restate our perspective on this, we want to learn facts about the training distribution and then, taking those facts as granted, mechanistically explain the behavior of our neural network.

  Roughly speaking we want the facts we need to learn to be as simple as possible. “The model M always behaves nicely when we ask it to” is a very complicated fact, about as complicated as the model M itself. Of course it’s entirely possible that the data distribution is “Sample randomly, but reject and sample again if M isn’t nice.” But that’s a very complicated hypothesis that no Bayesian or machine learning system would adopt.

  By comparison, “Obama” follows “Barack” is a simple regularity, which in fact is represented by a tiny set of parameters within your model. What we want to get is a giant conjunction of facts like this, corresponding in a very very rough sense to a subset of the model’s parameters (or a subset of the selection used to pick out the model), from which we can derive the performance of the model.

  So it’s not that we need to draw a line, it’s just that we prefer simpler facts. Sometimes there isn’t anything simpler to say—for example if you sample a function f at random and generate pairs (x, y=f(x)), there is no shorter explanation for the brute fact that y = f(x). But for models that are doing interesting internal work, which is a precondition for misalignment, there is all kinds of internal cognitive machinery—algorithms for planning, memory, inference, etc.---for which the behavior is a nontrivial computational fact. In these cases we think that we can separate out a smaller hypothesis about the data distribution.

  This isn’t ARC’s main focus right now, but we could test it pretty quickly once we are actually able to explain the behavior of neural networks in purely computational environments. I’ve spent a while thinking about it and I feel pretty good that we’ll be able to make something work. I don’t think this is a shell game, I think ARC is really focusing on the hard part of the problem and it’s not too worrying to set aside this issue until we see whether we succeed or fail at our core ambition.

  I think this concern looked a lot worse when we were considering treating some logical facts as empirical regularities, and when we were thinking about explanations as determined by our preferences and capacity allocation rather than an exhaustive account of a model’s learned structure. Of course those changes make our overall agenda look a lot more ambitious, but I think they shift the difficulty to the earlier steps of the agenda and make the rest of the path look a lot more plausible.

• paulfchristiano 5 Aug 2026 0:11 UTC

  19 points

  0

  in reply to: phoenix’s comment on: Re­turn­ing to ARC

  We’re focused on having an exhaustive fine-grained understanding, not just a rough approximation. We can’t yet succeed even for the simplest networks; explaining the AlgZoo models would be a huge milestone for us. (I don’t think any of the mechanistic interpretability approaches can handle any of these models either, but because we have a fairly precise sense of what we mean by “explain” it’s obvious that we’re not done yet.)

  Our focus is developing scalable methods that could be applied to the frontier. We don’t expect to ever shed unnecessarily fine detail,^[1] just to handle larger and larger models by spending more and more compute. If a method doesn’t scale as well as the training process itself then we’re prepared to give up on it.

  I also really like ambitious mechanistic interpretability, although my own biased view is that ARC’s agenda is the most promising approach to realizing that dream. I think both ARC’s agenda and traditional ambitious mechinterp are extremely hard and I understand why people are trying to do something more pragmatic.

  1. ^

     There is a subtle distinction here, where we want to capture all the learned structure in a model, but we think there is an exponential amount of unstructured detail in the behavior itself and in that setting we want to have algorithms that capture finer and finer details as we spend more and more compute.

Re­turn­ing to ARC

• paulfchristiano 17 Jun 2026 15:26 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: James Payor’s comment on: An­nounc­ing the ARC White-Box Es­ti­ma­tion Challenge

  I believe that all the issues you raised have been fixed, let us know if that’s not the case.

• paulfchristiano 13 Jun 2026 19:15 UTC

  28 points

  2

  in reply to: Yair Halberstadt’s comment on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  ARC works on theoretical alignment, but I think it’s reasonably clear how ARC’s work fits in with the current LLM paradigm (just apply our methods to LLMs!). It’s just an ambitious foundational project that has a very different risk/​reward profile from more incremental work. More incremental work naturally looks more and more exciting over time as AI improves (since the real-world problems get closer ad closer to your long-run concerns).

  There was an assumption 5 years ago that AGI requires fundamental insights into intelligence.

  I think a lot of people who have been working on theoretical alignment have long believed that LLMs could scale to AGI. Here’s me from 10 years ago:

  It now seems possible that we could build “prosaic” AGI, which can replicate human behavior but doesn’t involve qualitatively new ideas about “how intelligence works:”

  • It’s plausible that a large neural network can replicate “fast” human cognition, and that by coupling it to simple computational mechanisms — short and long-term memory, attention, etc. — we could obtain a human-level computational architecture.

  • It’s plausible that a variant of RL can train this architecture to actually implement human-level cognition. This would likely involve some combination of ingredients like model-based RL, imitation learning, or hierarchical RL.

  [...]

  I think that prosaic AGI should probably be the largest focus of current research on alignment.

• paulfchristiano 12 Jun 2026 19:26 UTC

  104 points

  31

  on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  There are many people working on aligning existing AI systems and understanding the alignment of existing AI systems. In my opinion many of the techniques and lessons from that work are likely to remain important for aligning increasingly powerful AI systems. For example, I care a lot about techniques for scalable supervision, broader scientific understanding of generalization (e.g. for honesty or reward hacking), mechanistic interpretability, behavioral red teaming, etc. In aggregate there are probably a few hundred people working on research that I’d classify as “direct alignment work,” which is fewer than you might think but still far from “almost nobody.”

  There has been much more investment in incremental improvements than indefinitely scalable methods, motivated partly by an increasing interest in aligning slightly superhuman systems and then “building the plane as we fly it.” I think it would be a definitional stretch to say that more incremental work doesn’t count as alignment, whether because you think it won’t scale indefinitely or because it could end up just training models that are more robustly incentivized to do what humans want them to do.

  (There has also been a big increase in methods for detecting and mitigating potential misalignment relative to building aligned AI systems, motivated partly by a belief that it will be easier to improve alignment once we have better examples of misalignment. I think it’s reasonable to make some distinction between that kind of research and efforts to directly make AI systems more aligned, though I think it’s counterintuitive and unnecessarily confrontational to say those people aren’t “working on alignment.” If you asked me what is the best way to “work on alignment” I might suggest developing model organisms of misalignment, and I think it’s generally very plausible that better scientific understanding and better measurement tools is most of the action.)

  I think a lot of people on this site are dismissive of all of those more incremental efforts and would say that “production alignment” is unrelated to the problem of aligning superhuman AI. I’ve spent a lot of time engaging with this community and find the standard arguments unpersuasive. I think I have a deep understanding of the conceptual difficulties for scalable alignment methods and in my view ARC is doing some of the most promising work for addressing those difficulties. But even understanding all of that I still think it would be a huge mistake to conclude that the incremental progress most people are investing in doesn’t help.

  I do think it’s great for people to make investments in foundations and indefinitely scalable methods. There’s a real chance that other methods will break down during an intelligence explosion, or that they will just never work particularly well. And I do think that machine learning researchers, policy makers, and the EA community are predictably underrating more scalable and foundational work. I just think the OP is a significant overstatement reflecting some significant unspoken assumptions.