Buck

• Buck 12 Jun 2026 12:17 UTC

  28 points

  14

  on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  I agree with your main point and I agree that this point seems curiously underrated by AI safety people.

  I don’t understand whether by “alignment” you mean:

  • Indefinitely scalable alignment: techniques that aim to be robust for AIs of any capability level

  • Alignment of any superintelligences: techniques that aim to work for the earliest superintelligent AIs. This might be much easier and is plausibly sufficient (as long as these AIs have enough time to develop techniques to align their successors).

  Either way, I think it probably makes sense to use a more specific term than “alignment” for this problem: it’s so natural to talk about whether non-superintelligent systems are aligned, and the alignment of non-superintelligence is IMO important for AI risk.

  As another note, “aligned with human values” is maybe pretty different from “follow human instructions”. I think ARC’s intended techniques are agnostic to whether their model is truly in its heart aligned, they just want to make models that follow their spec. So e.g. a model that is a paperclipper but will never act on its paperclipping urges would be fine.

  I don’t know why you think debate work at GDM counts as alignment if you don’t think that various other random prosaic alignment stuff counts. Debate is clearly not indefinitely scalable and in any case it is a technique for generating rewards, which doesn’t suffice for alignment unless you make dubious assumptions or use some other technique on top.

• Buck 9 Jun 2026 5:20 UTC

  4 points

  0

  in reply to: habryka’s comment on: The Problem

  This isn’t actually clear to me; is this just based on your impression? I’m not skeptical, I just don’t have an opinion.

Effi­cient trade­offs and the safety-use­ful­ness trade­off model

• Buck 3 Jun 2026 16:56 UTC

  18 points

  12

  in reply to: G Wood’s comment on: My fa­vorite de­pic­tion of utopia

  yeah, it’s definitely very scary! The future will likewise probably be very scary, and we have less reason to expect it to be benevolent.

• Buck 1 Jun 2026 4:15 UTC

  3 points

  0

  in reply to: Johan David Bonilla’s comment on: Notes on axes of vari­a­tion in third-party risk assessment

  Yeah. I agree that that tension is pretty concerning. I have read about Enron (as well as most of the other famous examples of corporate disasters) and am definitely somewhat inspired by that.

  I’ll note that Redwood is actually mostly not in the bind you describe; we aren’t very dependent on official relationships with AI developers; us getting blackballed from official relationships with them would be a bummer but by no means an existential threat.

Notes on axes of vari­a­tion in third-party risk assessment

• Buck 28 May 2026 1:03 UTC

  5 points

  1

  in reply to: Dan Braun’s comment on: Dan Braun’s Shortform

  I think I’m glad Steven Byrnes is doing what he’s doing, but notably he is in fact funded to do it! I don’t know how hard it was for him to get funding.

  I don’t really feel excited for people being more encouraged to do deep learning theory or agent foundations. I do appreciate that people tried out agent foundations back in the day.

• Buck 27 May 2026 18:36 UTC

  24 points

  5

  in reply to: Dan Braun’s comment on: Dan Braun’s Shortform

  I wish you had said ambitious AI safety research directions that you’re glad (at least ex ante) that people have pursued; it’s hard for me to otherwise know what I think of this.

  I am glad ex ante about ARC’s work, which counts. I think mech interp was probably pretty bad for the AI safety field overall (because of opportunity cost of people who worked on it), and I think its vibe of ambitiousness probably made the research go substantially worse (e.g. I wish people had either had the pragmatic interpretability attitude earlier, or that they’d been much more careful when making claims about their progress on ambitious goals). I can’t think of ambitious research directions that I wish were more strongly incentivized by the AI safety community; I think LWers are overall too enthusiastic and credulous about novel ambitious research directions.

  Unfortunately, job and funding incentives are biased against research ambition.

  I agree with job incentives (because many jobs are at AI companies that end up pushing people to contribute to the problems that are currently important), but I don’t really agree re funding incentives.

• Buck 27 May 2026 18:28 UTC

  25 points

  0

  on: Buck’s Shortform

  The duality between constrained and unconstrained optimization
  
  I use this concept all the time, but many rationalists don’t know it:
  
  When you’re solving a constrained optimization problem with nice properties [1]—for example, you’re playing a game where you buy continuous quantities of different resources at different prices and you want to maximize your probability of winning—you can rephrase the problem as an unconstrained optimization problem. In that example, the equivalent unconstrained optimization problem is “I want to maximize my probability of winning, and I have an unlimited budget, but every dollar I spend on resource reduces my probability of winning by x%.” for some x.
  
  To see why this is true, consider an optimal solution to your constrained optimization problem; let’s say you’re trying to maximize U(allocation). It must be the case that all resources[2] have the same utility returns to spending more on them, because otherwise you’d reduce spending on a resource with lower returns and spend the savings on a higher-return resource. So every resource has the same derivative of U wrt spending. This derivative is called the “shadow price”—it’s the exchange rate between utility and dollars. So if we had instead been solving the unconstrained optimization problem where you didn’t have a budget but your utility was U’(allocation) = U(allocation) - shadow_price * total_spending, we’d have picked the same solution.
  
  Of course, we don’t know what the shadow price is for a particular budget. So if you want to use this solution method to find an optimal allocation for a given budget, you need to basically guess and check different choices of shadow price until you find one that matches your budget—you pick a shadow price, solve for optimal allocation, then see how much you spent; if you spent too much, increase shadow price, if you spent too little, decrease it.
  
  This is useful because it’s great to have a concept of a shadow price—it’s way more intuitive to think in terms of a shadow price than a constraint, because it lets you think locally about one choice rather than needing to constantly think globally about all the other choices. Also, it’s often mathematically way simpler easier to solve unconstrained optimization problems than constrained ones (e.g. you can use calculus).
  
  As an example of how this comes up in an ML context, consider the problems:
  
  * Train a policy to have a maximum KL divergence from an original policy
  * Train a policy with a loss term for the KL divergance from an original policy
  
  Solutions to one problem are solutions to the other.
  
  [1] This works as long as your problem is convex, which lots of problems are; there are weaker versions of this that work under weaker conditions.
  [2] more precisely, all resources that you’re buying a non-zero amount of

How use­ful is the in­for­ma­tion you get from work­ing in­side an AI com­pany?

A re­view of “In­ves­ti­gat­ing the con­se­quences of ac­ci­den­tally grad­ing CoT dur­ing RL”

• Buck 7 May 2026 0:05 UTC

  8 points

  2

  in reply to: Linch’s comment on: Linch’s Shortform

  Have you read Schelling’s “Arms and Influence” (IIRC, chapter 1)? He argues that norms against killing civilians made a lot of sense for the couple of hundred years leading up to World War 2, but then stopped being as stable, hence strategic bombing.

• Buck 1 May 2026 22:03 UTC

  90 points

  33

  on: Buck’s Shortform

  AI safety people often worry about their work having capabilities externalities. I feel like the inverse problem is often a much bigger deal: the stuff you do might be done later by the capabilities/​product people in the course of their work, meaning your work isn’t counterfactual! I think people should try to distinguish these concerns more clearly.

  (See also Bronson Schoen’s similar post three hours earlier. I wrote this shortform in Redwood slack a few weeks ago and am just posting it now. Obviously this isn’t a very original take.)

• Buck 14 Apr 2026 23:36 UTC

  3 points

  0

  in reply to: sjadler’s comment on: sjadler’s Shortform

  in part because it might throw off other information they don’t necessarily want to.

  What other information does it release? All I can see is that it tells you how many agent trajectories they’re running. They could obfuscate that by also generating a bunch of hashes of random strings and publishing those too.

• Buck 14 Apr 2026 18:57 UTC

  2 points

  0

  on: At­tack Selec­tion In Agen­tic AI Con­trol Evals Can De­crease Safety

  Thanks for this.

  My understanding is that lots of control-ish evaluations, e.g. the SHADE-Arena and LinuxArena results reported in the Mythos alignment risk updates, do not incorporate attack selection. (Except I guess that LinuxArena filtered main tasks to be relatively good opportunities to attack...) Do you have a sense of how much of a problem this is quantitatively, or whether it invalidates the qualitative conclusions that are made using those evals?

• Buck 13 Apr 2026 21:22 UTC

  27 points

  24

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  I think that Scott’s recent tweets on this topic have actually been sort of unreasonable and impolite.

  LessWrong serves as the public square for the AI safety community

  Yes, if the proposal was to ban Scott from LessWrong I’d be much more opposed than if it was to (perhaps! this probably won’t even happen!) not host ACX meetups for free.

• Buck 13 Apr 2026 18:04 UTC

  41 points

  25

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  I think Habryka’s tweet there is totally reasonable.

  By default, I think it’s totally fine for people to use evidence and reason to make choices about which other people they want to support in various ways. This is the obvious default, and perhaps we don’t need to say anything more about it, but I’ll elaborate on what I see as the best relevant argument for why this would be bad, and explain why I don’t think it applies here:

  In some cases, people build infrastructure where they promise not to use the power they get from running the infrastructure in various ways. For example, it would be considered totally unacceptable for LessWrong or Facebook or whatever to post people’s private messages in order to accomplish their objectives. And I think it’s reasonable for users of Twitter to not like it if the algorithm is specifically designed to boost certain political views. As another example, MATS uses an external council to decide which mentors should be supported by MATS; they are explicitly trying to avoid consolidating a certain type of power so that other actors (in this case, maybe mostly funders) are happier to empower them in other ways.

  But that’s not what’s happening here. Hypothetically, Lightcone could have claimed that they would provide free space for any event that was approved of by a broad swathe of AI safety people, so that AI safety people were more willing to empower Lightcone. But that didn’t happen at all. Lightcone never said that. Oli is just using the authority that he has as the leader of an organization to take actions in a way that I think is totally consistent with the expectations he’s tried to set with his public communications.

• Buck 11 Apr 2026 14:30 UTC

  22 points

  19

  in reply to: RobertM’s comment on: Dario prob­a­bly doesn’t be­lieve in superintelligence

  Ok. Note that in most groups, worry about misuse risks (for a variety of definitions of misuse risks) is positively correlated rather than negatively correlated with your estimate of misalignment risk and how plausible you think it is that qualitative superintelligence is developed!

• Buck 11 Apr 2026 6:21 UTC

  44 points

  21

  on: Dario prob­a­bly doesn’t be­lieve in superintelligence

  Your understanding matches mine, based mostly on conversations with Anthropic staff about his beliefs.

  Dario has paid substantial political costs by making an enemy of the Trump administration on the subject of chip export controls (and then, later, Acceptable Use Policies, though this one is weaker because failing to hold the line here could plausibly have cost him more in terms of employee morale/​turnover than he lost from the DoW’s enmity). This is strongly consistent with being quite worried about misuse risks, particularly from authoritarian governments.

  This doesn’t seem very strongly relevant to whether he believes in qualitative superintelligence. It’s definitely the case that he is way more worried about misuse risks than AI takeover.

• Buck 10 Apr 2026 23:05 UTC

  5 points

  2

  in reply to: Shankar Sivarajan’s comment on: ClaireZa­bel’s Shortform

  I seriously think that lots of the decision makers were wrong. See e.g. Yglesias’s article about it.