Buck

• Buck 10 Sep 2026 5:25 UTC

  10 points

  8

  in reply to: habryka’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  Paul probably did serve as a substantial public counterweight to Yudkowsky-style pessimism within the AI-safety community

  This seems right but isn’t how I interpreted Ben’s claim; I interpreted Ben as saying that Paul was a “optimistic counterweight” in the broader public discourse about AI safety (which is who one would naturally “sound the alarm” to), which is obviously different than the question of how he fit into the discussion within the AI safety community!

• Buck 7 Sep 2026 19:04 UTC

  14 points

  11

  on: Let’s talk about the AI co­or­di­na­tion problem

  I agree that it would be a mistake to assume that this coordination is impossible, but I think your “ending wars” example is less rosy than you make it out to be. In particular, wars famously sometimes happen for a while before they end, even though at every moment the war is usually worse than a negotiated settlement. There are various explanations for this (my favorites are Fearon, and Schelling “Strategy of Conflict” and some of “Arms and Influence”). I think that the a priori position should be that mutually damaging failures to coordinate sometimes happen and sometimes don’t.

• Buck 7 Sep 2026 1:30 UTC

  12 points

  4

  in reply to: maxnadeau’s comment on: Alex Mallen’s Shortform

  I suspect what’s going on here is that you’re using the term “Constellation” to refer almost entirely to Redwood

  I think it’s actually even narrower than this: I think Alex isn’t trying to talk about e.g. me, Ryan, or himself, he’s talking about more junior staff. As he notes, Ryan and I have actually been very interested in these other things, and I think that our writing, talks, and public statements have been totally reasonable on the topics he’s saying were underrated by “Constellation”.

• Buck 9 Aug 2026 21:23 UTC

  22 points

  4

  in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform

  On a few occasions, I’ve tried to give agents ways of contacting me if they need help with something or if they think they’ve run across a problem in their environment. My experience is that if you tell agents they can do this, they will constantly spam you with irrelevant, dumb stuff. For example, whining that the environment must be bugged because their code isn’t working, when obviously what’s going on is that their code is wrong.

  Current LLMs might be better at this. And obviously you could train them to be better at not wasting your time with irrelevant messages.

The OpenAI/​Hug­ging­face in­ci­dent | Red­wood Re­search pod­cast epi­sode 2

• Buck 14 Jul 2026 1:18 UTC

  14 points

  0

  in reply to: habryka’s comment on: jelly’s Shortform

  I’m very sold that it makes sense to talk to the cake size champion in exactly the scenario he described. Throughout that document (which I love and have posted repeatedly to Redwood slack) the examples are constantly insane, and it’s hard for me to tell how much that’s because Mr Beast productions are actually insane and how much it’s because he’s using weird examples.

• Buck 13 Jul 2026 17:20 UTC

  69 points

  18

  on: Buck’s Shortform

  People keep proposing that I should do a public debate with @Neel Nanda on “is mech interp valuable”. But I think me and Neel don’t really disagree on very much on the topic anymore. I largely agree with his position on interpretability articulated in “A Pragmatic Vision for Interpretability” and elsewhere, and both of us are pretty sympathetic to each other’s research strategy and prioritization.
  
  I think Neel and I genuinely disagreed from 2023 to mid 2025, but by the end of that time period we had pretty similar positions, mostly due to Neel coming around to my position on some broad questions about interp and me feeling happier with the quality of interp research (thanks substantially to Neel).

• Buck 12 Jul 2026 20:45 UTC

  32 points

  7

  in reply to: Eva Lu’s comment on: evalu’s Shortform

  I think this is directionally right, though it’s worth noting that the EAs and GiveWell people also contributed their own culture which also has very good epistemics by normal standards; IMO the EAs/​GiveWell people have epistemic practices that added useful ingredients to the epistemic culture around here. (This is consistent with your text if that’s who you mean by “everyone who was around in the old days”.)

  Also I think a lot of this effect is a selection pressure for people who are particularly altruistic and interested in good epistemics, rather than causing people to become more reasonable (though the latter also happened)

• Buck 5 Jul 2026 23:31 UTC

  9 points

  3

  in reply to: Taylor G. Lunt’s comment on: Tay­lor G. Lunt’s Shortform

  I would be very excited for you to write up the examples that make you feel this way; I’m very unsure about it.

• Buck 5 Jul 2026 23:30 UTC

  5 points

  0

  in reply to: J Bostock’s comment on: Jemist’s Shortform

  If a result doesn’t generalize between optimizers, that often is a reason I’m much less interested in it, because this is evidence against it generalizing to future situations I care about. This is especially true for research that tries to learn facts, rather than research that tries to produce models with particular properties so that they can be used as subjects of research. E.g. it’s very hard to make sandbagging model organisms, and it would be sort of handy to have those model organisms around, so it is useful to know whether different optimizers help (my coworkers looked into this and didn’t make it work.)

• Buck 5 Jul 2026 20:53 UTC

  16 points

  0

  on: I think al­ign­ment work is more promis­ing than con­trol work

  Solid post. (Alec and I discussed this over a call, this is me summarizing my state after that.)

  I suspect that I think there should be more investment in control as a proportion of AI safety research effort than Alec thinks, though it’s hard to operationalize well enough to tell. My guesses about some interesting disagreements:

  • As Fabien said, I think we’d need to talk in more detail about what research projects specifically you’re excited for in order to do this analysis.

    • I think that Alec is a lot more excited about various alignment research projects than I am.

  • I’m a lot more pessimistic about alignment of current and future models than Alec is.

  • I’m more excited about the possibility that you can iterate on control measures better than alignment methods at crunch time.

• Buck 5 Jul 2026 20:49 UTC

  2 points

  0

  in reply to: Alec Harris’s comment on: I think al­ign­ment work is more promis­ing than con­trol work

  Sandbagging is maybe really hard though! See here. In general, the possibility of running control evals at crunch time is a lot of why I like control.

• Buck 5 Jul 2026 19:02 UTC

  5 points

  0

  in reply to: Alec Harris’s comment on: I think al­ign­ment work is more promis­ing than con­trol work

  (Moving back up to the top of your bullets) It seems to me like there are a bunch of cursed things about trying to know if you are making progress on control, such as:
  - A scheming AI may come up with strategies very dissimilar to your red-team
  - The environment may look super different with weird changes in scaffolding, coding language, etc.
  - The models themselves might be very different and this might change things (maybe continual learning could make collusion easier, etc.)
  (This is along with smaller, more basic disanalogies like maybe adding coding backdoors is a bit different than trying to have a rogue internal deployment or something.)

  A lot of what excites me about control is that you can, at crunch time, run those experiments in the actual setting you care about. This means that a lot of our work now should be focused on prepping to run those experiments.

• Buck 3 Jul 2026 17:57 UTC

  11 points

  4

  on: An­nounc­ing the Safe Pareto Im­prove­ments (SPI) Fun­da­men­tals Program

  I’m excited for this! I think that SPIs are one of the top few research directions that mathematically inclined people can contribute to to make AI go better.

• Buck 12 Jun 2026 12:17 UTC

  33 points

  16

  on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  I agree with your main point and I agree that this point seems curiously underrated by AI safety people.

  I don’t understand whether by “alignment” you mean:

  • Indefinitely scalable alignment: techniques that aim to be robust for AIs of any capability level

  • Alignment of any superintelligences: techniques that aim to work for the earliest superintelligent AIs. This might be much easier and is plausibly sufficient (as long as these AIs have enough time to develop techniques to align their successors).

  Either way, I think it probably makes sense to use a more specific term than “alignment” for this problem: it’s so natural to talk about whether non-superintelligent systems are aligned, and the alignment of non-superintelligence is IMO important for AI risk.

  As another note, “aligned with human values” is maybe pretty different from “follow human instructions”. I think ARC’s intended techniques are agnostic to whether their model is truly in its heart aligned, they just want to make models that follow their spec. So e.g. a model that is a paperclipper but will never act on its paperclipping urges would be fine.

  I don’t know why you think debate work at GDM counts as alignment if you don’t think that various other random prosaic alignment stuff counts. Debate is clearly not indefinitely scalable and in any case it is a technique for generating rewards, which doesn’t suffice for alignment unless you make dubious assumptions or use some other technique on top.

• Buck 9 Jun 2026 5:20 UTC

  4 points

  0

  in reply to: habryka’s comment on: The Problem

  This isn’t actually clear to me; is this just based on your impression? I’m not skeptical, I just don’t have an opinion.

Effi­cient trade­offs and the safety-use­ful­ness trade­off model

• Buck 3 Jun 2026 16:56 UTC

  20 points

  25

  in reply to: G Wood’s comment on: My fa­vorite de­pic­tion of utopia

  yeah, it’s definitely very scary! The future will likewise probably be very scary, and we have less reason to expect it to be benevolent.

• Buck 1 Jun 2026 4:15 UTC

  4 points

  0

  in reply to: Johan David Bonilla’s comment on: Notes on axes of vari­a­tion in third-party risk assessment

  Yeah. I agree that that tension is pretty concerning. I have read about Enron (as well as most of the other famous examples of corporate disasters) and am definitely somewhat inspired by that.

  I’ll note that Redwood is actually mostly not in the bind you describe; we aren’t very dependent on official relationships with AI developers; us getting blackballed from official relationships with them would be a bummer but by no means an existential threat.

Notes on axes of vari­a­tion in third-party risk assessment