paulfchristiano

• paulfchristiano 5 Aug 2026 4:22 UTC

  LW: 22 AF: 11

  3

  AF

  in reply to: habryka’s comment on: Re­turn­ing to ARC

  By saying these techniques “break down” I mean “they cannot be used to get competitive work out of an AI system without having it take over.” I believe that:

  1. Existing AI systems won’t take over. I think there are still a few lines of defense before existing AI systems pose a significant risk of takeover (though it is not clear how long these will last).

  2. Modern AI systems are not pushing the limits of existing methods. I think you could push those methods much harder in order to significantly reduce the probability of takeover.

  In my mind that latter point is one of the main arguments against working on a project like ARC. I think it’s fairly likely that we live in one of the 70-80% of worlds where existing methods can in principle scale to broadly superhuman AI, but that we still get an AI takeover because our implementation isn’t good enough.

• paulfchristiano 5 Aug 2026 1:23 UTC

  10 points

  1

  in reply to: David Johnston’s comment on: Re­turn­ing to ARC

  Your reading makes sense and I don’t want to quibble over semantics (or foreclose potentially productive dialog). I do think there’s an important distinction between:

  1. Optimize a loss function that captures performance on the training distribution, and then shape the architecture /​ curriculum /​ regularizer /​ etc. so that the model behaves in a desirable way out of distribution.

  2. Define a loss function that captures performance off of the training distribution (e.g. what would happen in conditionals that we can’t sample in the lab) and optimize that.

  I was describing ARC as trying to pursue option #2.

  That said, in reality we are going to need to characterize the real world using learned hypotheses, since we can’t give a formal characterization without using ML. And then of course we care about how those hypotheses generalize. So regardless of the distinction above I do think it’s fair to characterize our work as an example of “understanding and shaping ML generalization.” We’re just willing to make much larger changes to get the generalization we want. In particular we are not trying to understand or shape how an agent generalizes, but instead are trying to separate the learning of hypotheses about the data (which do need to generalize well) from mechanistic analyses of predictors and agents that behave well under those hypotheses.

  What I really care about is that our methods would likely work even if competitive RL converges robustly to training-gaming behavior, while I’m claiming that other approaches rest on the empirical bet that what generalization you get is sensitive to some training parameters that we can tweak without big performance hits.

• paulfchristiano 5 Aug 2026 1:06 UTC

  5 points

  0

  on: Ob­sta­cles in ARC’s agenda: Find­ing explanations

  Coming back to this (now that I’m back at ARC) I think we have much clearer answers to most of these questions in 2026 than we did in 2025. I don’t know if these answers will necessarily be more comforting to you but they are at least much bolder: less ambiguous, more falsifiable. I think that’s good news for us either succeeding or failing faster.

  once we acknowledge that there are phenomena within a neural net that can’t and shouldn’t be explained, it’s unclear what our exact goal is and where we draw the line, what can be classified as an empirical regularity and what needs to be further explained.

  At the time David wrote this we were imagining that some logical facts (like “a queen is worth 9 pawns”) would get treated as inexplicable empirical regularities. That was scary, because if you treat the value of a queen as a brute empirical fact then why don’t you treat “the model tells the truth” as a brute empirical fact?

  We no longer believe that. Now we think that in a purely computational system we explain everything. We only need to deal with empirical regularities when we have data drawn from an unknown input distribution. So e.g. we should be able to mechanistically predict that a model is 96.2% less likely to win a chess game if it loses its queen (given that its playing against a particular opponent), with accuracy strictly greater than simply playing out a bunch of games to check.

  There is still real work in distinguishing “learning about the data distribution” from “testing computational machinery.” (I’ll return to this at the bottom of the comment.) But the conceptual line between them is now pretty clear and explaining purely computational systems is now a clean subproblem. In particular, if we fail at it we won’t be able to throw up our hands and say “it’s all an empirical regularity.”

  In ARC’s current thinking, the plan is not to have a clear line of separation between things we need to explain and things we don’t. Instead, the loss function of explanation quality should capture how important various things are to explain, and the explanation-finding algorithm is given a certain compute budget to build up the explanation of the model behavior by bits and pieces. [...] my main position is just that solving capacity allocation is crucial for explaining anything interesting at all, and it seems extremely hard, and we don’t know how to do it.

  David was going a bit out on a limb here since ARC didn’t really have a clear view in 2025. But fortunately in the last year the plan has gotten much more concrete, and it looks like we don’t need anything like capacity allocation.

  Back then we imagined an estimator G(X|pi) that estimates a quantity X given an argument pi.

  Now we imagine a family of estimators G_m(X|pi) parametrized by the computational budget m. If the quantity X is chosen randomly then pi can be empty and the estimates G_m(X) should be accurate (that’s what happens when we evaluate random neural networks). If instead X is highly structured then pi is suppsoed to “explain” that structure. If we have a structured neural network with a trillion parameters, we are hoping for a single explanation pi with roughly a trillion parameters that lets us answer random questions about that neural network, in the same way that the unconditional G_m(X) is good for random quantities X.

  (That’s a tiny bit too crude—we want a pi that lets us answer typical questions, but if we pick out a very specific question of n bits then we may need to supply another n bits of advice to complement the trillion parameters of pi. You can see the rough formalization for this here.)

  The upshot is that we aren’t picking pi to explain specific questions, we are learning it over the course of training to make sense of the model itself. We acknowledge that there is an infinite list of unimportant unstructured details, but those are now absorbed into the work done as G_m(X) spends more time thinking about X, rather than into pi itself. There are still conceptual issues here as we combine this with empirical regularities, but the situation is much cleaner than it was last year.

  We can’t really say we are fully explaining why the AI has high accuracy on a set of tasks, as the explanation would need to involve facts about the outside tools (like the internet) the AI is using. If we are only allowed to refer to the things the AI believes about the world, is our explanation about why the AI believes it will have a high accuracy?

  Our goal is explaining why the AI has high accuracy on a set of tasks. The explanation needs to involve facts about the environment, including e.g. “When I type a mathematical expression into the calculator, I get an accurate result.” We are going to represent those facts about the environment using the same kinds of flexible representations that we are using in ML. The loss we use for those facts will be a prediction loss, though rather than predicting bits we will be predicting various intermediate quantities.

  Here we do face a capacity allocation question, not in training explanations but in deciding what facts about the data distribution to model. We’d like to model roughly the same facts the AI does. The AI “learns” things by gradient descent minimizing the loss, and we think that if we learned everything about the environment that is needed to predict the loss then that will capture (at least) everything that the model could have learned about the environment.

  This area hasn’t gotten as much love as the prior two questions, but I feel pretty good about it overall. I think if we’re going to fail, we’ll probably also fail to explain how and why neural networks work in well-defined computational environments (like a Python interpreter).

  Similarly, if the vault-protecter AI is allowed to design a new AI, or just hire humans to do some tasks, it is very dangerous to rely on teleological explanations of the new AI or humans following the goals given to them, as you still can’t distinguish if they are achieving the goal through legitimate means.

  I agree! If your AI builds another AI to do the job, then you will be uncertain about whether the new AI is aligned and how it will generalize. Your AI will also be uncertain about whether the new AI is aligned.

  ARC’s position on this is that there’s no guarantee that an aligned AI will solve alignment of the further AI systems it builds (just like there’s no guarantee that it will successfully handle any of the other challenges it faces). ARC will try to help the automated AI researchers in exactly the same way we try to help the human AI researchers (i.e. by coming up with great alignment techniques it can use), and AI companies will try to help the automated AI researchers by making them smarter and wiser. But this is not the kind of agenda that gives you a promise that all future AI systems will be aligned, it’s the kind of agenda that tells you how to align AI systems in the current paradigm.

  ---

  I want to talk about empirical regularities one more time since it’s a recurring theme in this post:

  This means we will need to draw the line somewhere between things that are obviously unexplainable (‘Barack’ followed by ‘Obama’), and things that are complex enough that catastrophic deceptive alignment can hide inside the observed phenomenon (the model always behaves nicely when we ask it to).

  To restate our perspective on this, we want to learn facts about the training distribution and then, taking those facts as granted, mechanistically explain the behavior of our neural network.

  Roughly speaking we want the facts we need to learn to be as simple as possible. “The model M always behaves nicely when we ask it to” is a very complicated fact, about as complicated as the model M itself. Of course it’s entirely possible that the data distribution is “Sample randomly, but reject and sample again if M isn’t nice.” But that’s a very complicated hypothesis that no Bayesian or machine learning system would adopt.

  By comparison, “Obama” follows “Barack” is a simple regularity, which in fact is represented by a tiny set of parameters within your model. What we want to get is a giant conjunction of facts like this, corresponding in a very very rough sense to a subset of the model’s parameters (or a subset of the selection used to pick out the model), from which we can derive the performance of the model.

  So it’s not that we need to draw a line, it’s just that we prefer simpler facts. Sometimes there isn’t anything simpler to say—for example if you sample a function f at random and generate pairs (x, y=f(x)), there is no shorter explanation for the brute fact that y = f(x). But for models that are doing interesting internal work, which is a precondition for misalignment, there is all kinds of internal cognitive machinery—algorithms for planning, memory, inference, etc.---for which the behavior is a nontrivial computational fact. In these cases we think that we can separate out a smaller hypothesis about the data distribution.

  This isn’t ARC’s main focus right now, but we could test it pretty quickly once we are actually able to explain the behavior of neural networks in purely computational environments. I’ve spent a while thinking about it and I feel pretty good that we’ll be able to make something work. I don’t think this is a shell game, I think ARC is really focusing on the hard part of the problem and it’s not too worrying to set aside this issue until we see whether we succeed or fail at our core ambition.

  I think this concern looked a lot worse when we were considering treating some logical facts as empirical regularities, and when we were thinking about explanations as determined by our preferences and capacity allocation rather than an exhaustive account of a model’s learned structure. Of course those changes make our overall agenda look a lot more ambitious, but I think they shift the difficulty to the earlier steps of the agenda and make the rest of the path look a lot more plausible.

• paulfchristiano 5 Aug 2026 0:11 UTC

  10 points

  0

  in reply to: phoenix’s comment on: Re­turn­ing to ARC

  We’re focused on having an exhaustive fine-grained understanding, not just a rough approximation. We can’t yet succeed even for the simplest networks; explaining the AlgZoo models would be a huge milestone for us. (I don’t think any of the mechanistic interpretability approaches can handle any of these models either, but because we have a fairly precise sense of what we mean by “explain” it’s obvious that we’re not done yet.)

  Our focus is developing scalable methods that could be applied to the frontier. We don’t expect to ever shed unnecessarily fine detail,^[1] just to handle larger and larger models by spending more and more compute. If a method doesn’t scale as well as the training process itself then we’re prepared to give up on it.

  I also really like ambitious mechanistic interpretability, although my own biased view is that ARC’s agenda is the most promising approach to realizing that dream. I think both ARC’s agenda and traditional ambitious mechinterp are extremely hard and I understand why people are trying to do something more pragmatic.

  1. ^

     There is a subtle distinction here, where we want to capture all the learned structure in a model, but we think there is an exponential amount of unstructured detail in the behavior itself and in that setting we want to have algorithms that capture finer and finer details as we spend more and more compute.

Re­turn­ing to ARC

• paulfchristiano 17 Jun 2026 15:26 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: James Payor’s comment on: An­nounc­ing the ARC White-Box Es­ti­ma­tion Challenge

  I believe that all the issues you raised have been fixed, let us know if that’s not the case.

• paulfchristiano 13 Jun 2026 19:15 UTC

  27 points

  2

  in reply to: Yair Halberstadt’s comment on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  ARC works on theoretical alignment, but I think it’s reasonably clear how ARC’s work fits in with the current LLM paradigm (just apply our methods to LLMs!). It’s just an ambitious foundational project that has a very different risk/​reward profile from more incremental work. More incremental work naturally looks more and more exciting over time as AI improves (since the real-world problems get closer ad closer to your long-run concerns).

  There was an assumption 5 years ago that AGI requires fundamental insights into intelligence.

  I think a lot of people who have been working on theoretical alignment have long believed that LLMs could scale to AGI. Here’s me from 10 years ago:

  It now seems possible that we could build “prosaic” AGI, which can replicate human behavior but doesn’t involve qualitatively new ideas about “how intelligence works:”

  • It’s plausible that a large neural network can replicate “fast” human cognition, and that by coupling it to simple computational mechanisms — short and long-term memory, attention, etc. — we could obtain a human-level computational architecture.

  • It’s plausible that a variant of RL can train this architecture to actually implement human-level cognition. This would likely involve some combination of ingredients like model-based RL, imitation learning, or hierarchical RL.

  [...]

  I think that prosaic AGI should probably be the largest focus of current research on alignment.

• paulfchristiano 12 Jun 2026 19:26 UTC

  104 points

  31

  on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  There are many people working on aligning existing AI systems and understanding the alignment of existing AI systems. In my opinion many of the techniques and lessons from that work are likely to remain important for aligning increasingly powerful AI systems. For example, I care a lot about techniques for scalable supervision, broader scientific understanding of generalization (e.g. for honesty or reward hacking), mechanistic interpretability, behavioral red teaming, etc. In aggregate there are probably a few hundred people working on research that I’d classify as “direct alignment work,” which is fewer than you might think but still far from “almost nobody.”

  There has been much more investment in incremental improvements than indefinitely scalable methods, motivated partly by an increasing interest in aligning slightly superhuman systems and then “building the plane as we fly it.” I think it would be a definitional stretch to say that more incremental work doesn’t count as alignment, whether because you think it won’t scale indefinitely or because it could end up just training models that are more robustly incentivized to do what humans want them to do.

  (There has also been a big increase in methods for detecting and mitigating potential misalignment relative to building aligned AI systems, motivated partly by a belief that it will be easier to improve alignment once we have better examples of misalignment. I think it’s reasonable to make some distinction between that kind of research and efforts to directly make AI systems more aligned, though I think it’s counterintuitive and unnecessarily confrontational to say those people aren’t “working on alignment.” If you asked me what is the best way to “work on alignment” I might suggest developing model organisms of misalignment, and I think it’s generally very plausible that better scientific understanding and better measurement tools is most of the action.)

  I think a lot of people on this site are dismissive of all of those more incremental efforts and would say that “production alignment” is unrelated to the problem of aligning superhuman AI. I’ve spent a lot of time engaging with this community and find the standard arguments unpersuasive. I think I have a deep understanding of the conceptual difficulties for scalable alignment methods and in my view ARC is doing some of the most promising work for addressing those difficulties. But even understanding all of that I still think it would be a huge mistake to conclude that the incremental progress most people are investing in doesn’t help.

  I do think it’s great for people to make investments in foundations and indefinitely scalable methods. There’s a real chance that other methods will break down during an intelligence explosion, or that they will just never work particularly well. And I do think that machine learning researchers, policy makers, and the EA community are predictably underrating more scalable and foundational work. I just think the OP is a significant overstatement reflecting some significant unspoken assumptions.

• paulfchristiano 4 Jun 2026 15:00 UTC

  5 points

  1

  in reply to: phoenix’s comment on: An­nounc­ing the ARC White-Box Es­ti­ma­tion Challenge

  I think the only difference between the phases is how locked in the rules are. During the warm-up round we may make big changes while we uncover bugs. In phase 1 we may make smaller changes. In phase 2 we won’t make changes so that people know how they are being graded.

  (AICrowd has run a lot of competitions and ARC has run none so we just handed it off to them and letting them make these kinds of decisions.)

• paulfchristiano 4 Jun 2026 14:03 UTC

  2 points

  0

  in reply to: Daniel Tan’s comment on: An­nounc­ing the ARC White-Box Es­ti­ma­tion Challenge

  Your score is the budget-adjusted MSE. You can see how it looks on the leaderboard, where the rank is by “adjusted score” which is 0.1-1x the “final layer MSE.”

  scoring-model.md is wrong or at least confusing, I’ll pass on the feedback. (I think maybe that’s because that document is just describing the “scoring model,” which is the input into the leaderboard score, but regardless it’s a misleading way to describe what’s going on.)

• paulfchristiano 3 Jun 2026 16:25 UTC

  13 points

  2

  in reply to: Guillaume Martres’s comment on: An­nounc­ing the ARC White-Box Es­ti­ma­tion Challenge

  I definitely agree that trained models are the real goal.

  One view we hold is that all the complexities and messiness of randomly initialized models are still present in trained models—just combined with a lot of richer structure, different sets of features that are relevant, and so on. Moreover, we think that current algorithms for randomly initialized networks are not yet good enough that you should expect it to be possible to analyze trained models in a convincing way, except perhaps in the extremely overtrained regime where the model converges to a highly simplified algorithm.

  We think there probably do exist adequate algorithms for analyzing random networks and the reason we haven’t found them is because so little research has addressed the question head on. And if that’s not possible we want to learn sooner and give up on the project.

  In my opinion you can see this issue a lot in mechanistic interpretability research. Often at some point an investigation into a particular behavior needs to throw up its hands and say “and then a bunch of messy and complicated stuff happens” and you are left wondering whether you understand the phenomenon or whether the important action is in all the messy details. (Except in the super overtrained setting.)

  I think that interpretability research would be more promising if we had a really solid understanding of “random messy stuff,” such that you could say “yes we understand the randomly initialized model” and then focus on whether you are able to keep handling new structure as it gets introduced by training. Rather than starting off confused and hoping that you can catch some glimpses of what training is doing despite the background confusion.

  (Of course in this comment I’m using a much more limited notion of “understanding” than interpretability researchers typically have in mind—I just want algorithms that are able to leverage a mechanistic analysis of the weights to answer concrete questions about the model. We hope that’s a productive substitution that can drive much faster progress and help us find scalable algorithms, and we think that this more limited sense would still be sufficient to solve alignment.)

An­nounc­ing the ARC White-Box Es­ti­ma­tion Challenge

Mechanis­tic es­ti­ma­tion for ex­pec­ta­tions of ran­dom products

• paulfchristiano 26 Apr 2026 19:36 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: JesseClifton’s comment on: Lukas Fin­nve­den’s Shortform

  If “managing the news” just means “making a decision in situation X such that you are glad to hear the news that you made that decision in situation X,” then I agree that’s a description of EDT. I think it’s a priori reasonable to manage news about what you decide to do, so I don’t see this as a fundamental reason that EDT is problematic. I usually associate the phrase with various intuitive mistakes that EDT might make, and then I want to discuss concrete cases (like Lukas’) in which it appears an agent did something wrong.

• paulfchristiano 26 Apr 2026 15:55 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: JesseClifton’s comment on: Lukas Fin­nve­den’s Shortform

  Are there cases where EDT manages the news for reasons other than anthropic updating? I’m not aware of any, and if not then it this is exactly because of the interaction of EDT and anthropic updating.

• paulfchristiano 26 Apr 2026 15:51 UTC

  LW: 4 AF: 3

  2

  AF

  in reply to: Lukas Finnveden’s comment on: Lukas Fin­nve­den’s Shortform

  In your example, you know T1 and your counterpart knows T2. You see your behavior as correlated with the behavior of your counterpart. Under these conditions, it seems like T1 can’t possibly be so fundamental that you need it in order to do EDT-style reasoning (otherwise your counterpart couldn’t). So even if you grant that you need some beliefs to get EDT off the ground, it seems like those beliefs must be in the intersection of T1 and T2, in which case you wouldn’t run into this problem.

  I think there are a lot of pretty basic open questions about how logically bounded minds work, especially if they want them to be philosophically principled all the way down. That’s confusing enough that it’s totally plausible that you’ll come out of it with some TDT-like story, where causality is a basic cognitive ingredient that makes the bootstrapping work at all. My guess is that in the end causality will just be a helpful language for talking about certain kinds of independence relationships rather than a fundamental cognitive building block or something with direct normative implications, but who knows.

  (Note for observers: I’m not expecting us to resolve any of those questions as part of AI safety.)

• paulfchristiano 15 Apr 2026 14:48 UTC

  11 points

  2

  in reply to: papetoast’s comment on: Self-driv­ing car bets

  Waymo’s website says they are serving riders in 11 US cities (all of which are among the 50 largest).

  Waymo feels very real in SF: it’s a product that’s easy to access, people use it because they like it rather than for novelty, it’s able to serve most routes within the city, and it has nearly as much market share as Uber (here’s some slightly stale data, I think by now its share is more like 25%). I don’t have direct visibility into the other 10 cities, but I think at least a few of them still have a waitlist. My current best guess is that we’ll hit my 2.5 year prediction a bit before 3 years (and probably hit my 3.5 year prediction a bit early).

  Overall I think that self-driving deployment has been faster than most technical people expected in 2016. It was probably somewhere around my 20th percentile.

• paulfchristiano 14 Apr 2026 17:52 UTC

  4 points

  2

  in reply to: habryka’s comment on: No77e’s Shortform

  When you wrote:

  Ajeya from a few years ago also said things more like this (more point 1 and 3, less point 2)

  I interpreted you as claiming that Ajeya had said “things more like:”

  In general, people worried about AI risk should coordinate as much as possible to play down our concerns, so as not to look like alarmists. This is very important in order to build allies and accumulate political influence, so that we’re well-positioned to act if and when an important opportunity arises.

  I don’t recall any examples of Ajeya saying or implying anything at all like that. I asked ChatGPT to try to find examples and I think it didn’t find anything.

  In your ChatGPT session, a typical example it cites is:

  In the AXRP discussion, she also says there were concerns that making the report seem too slick or official could increase capabilities interest.

  I think those examples don’t meaningfully support the original claim, at least as a typical reader would understand it.

• paulfchristiano 14 Apr 2026 16:03 UTC

  17 points

  1

  in reply to: habryka’s comment on: No77e’s Shortform

  Ajeya from a few years ago also said things more like this (more point 1 and 3, less point 2).

  I don’t remember anything like this. I think it might be misremembered or a strained interpretation.

  Here are points 1 and 3 for reference:

  1. AI is going to become vastly superhuman in the near future; but being a good scientist means refusing to speculate about the potential novel risks this may pose. Instead, we should only expect risks that we can clearly see today, and that seem difficult to address today.

  3. In general, people worried about AI risk should coordinate as much as possible to play down our concerns, so as not to look like alarmists. This is very important in order to build allies and accumulate political influence, so that we’re well-positioned to act if and when an important opportunity arises.

  I asked ChatGPT to read bioanchors (where I thought this was most likely to occur), and then to read all of her other writings looking for anything that fits that mode. Here’s its reply, not finding anything.

  The closest match it finds is that Ajeya often caveats her claims. For example from bio anchors:

  This is a work in progress and does not represent Open Philanthropy’s institutional view […] Accordingly, we have not done an official publication or blog post, and would prefer for now that people not share it widely in a low bandwidth way.

  I don’t think this matches points 1 or 3 well.

• paulfchristiano 8 Apr 2026 15:27 UTC

  81 points

  30

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  Speaking for myself, I’d say we’ve ruled out the most pessimistic scenarios I was taking seriously 15 years ago. I’ve always thought alignment would probably be fine, but conditional on not being fine there was a reasonable chance we’d have seen serious problems by now and we haven’t. On balance I’m more pessimistic than I was back then, but that’s because we’ve ruled out many more of the most optimistic scenarios (back then it wasn’t even obvious we’d be training giant opaque neural network agents using RL, that was just a hypothetical scenario that seemed plausible and particularly worrying!).

  If we want to go by Eliezer’s public writing rather than my self-reports, in 2012 he appeared to take some very pessimistic hypotheses seriously, including some that I would say are basically ruled out. For example see this exchange where he wrote:

  It’s not unthinkable that a non-self-modifying superhuman planning Oracle could be developed with the further constraint that its thoughts are human-interpretable, or can be translated for human use without any algorithms that reason internally about what humans understand, but this would at the least be hard.
  
  [...]
  
  It turns out that getting an AI to understand human language is a very hard problem, and it may very well be that even though talking doesn’t feel like having a utility function, our brains are using consequential reasoning to do it. Certainly, when I write language, that feels like I’m being deliberate. It’s also worth noting that “What is the effect on X?” really means “What are the effects I care about on X?” and that there’s a large understanding-the-human’s-utility-function problem here. [...] Does the AI know that music is valuable? If not, will it not describe music-destruction as an “effect” of a plan which offers to free up large amounts of computer storage by, as it turns out, overwriting everyone’s music collection?

  It seems like Eliezer is taking seriously the possibility that “describe plans and their effects to humans” requires the kind of consequentialism that might result in takeover, and that AI might be dangerous at a point when “understanding the human’s utility function” (in order to understand what effects are worth mentioning explicitly) is still a hard problem. Those look much less plausible now—we have AI systems that are superhuman in some respects and whose chains of thought are interpretable (for now) because they are anchored to cognitive demonstrations from humans rather than because of consequentialist reasoning about how to communicate with humans.

  This isn’t to say that concern is discredited. Indeed today we have AI systems that clearly know about our preferences, but will ignore them when it’s the easiest way to get reward. Chain of thought monitorability is possible but on shaky ground. That said, I think we’re ruling out plenty of even worse scenarios.