Buck

• Buck 9 Mar 2026 3:30 UTC

  6 points

  7

  in reply to: aysja’s comment on: Pod­cast: Jeremy Howard is bear­ish on LLMs

  I agree that the AIs are pretty bad at handling conceptually confusing stuff. I basically think of them as being incredibly knowledgeable, not that smart, and having huge amounts of intuition on how to program (mostly due to their knowledge and their having read huge amounts of code).

  My guess is that for any reasonable operationalization of “raw intelligence”, they’re getting smarter?

An­nounc­ing Con­trolConf 2026

• Buck 25 Feb 2026 6:45 UTC

  22 points

  6

  in reply to: habryka’s comment on: Re­spon­si­ble Scal­ing Policy v3

  Different people went different paths, but most of the ecosystem’s resources went into the latter kind of plan, with the two central pillars being to start and invest in evaluation companies like METR and Apollo (to develop evaluations and capability measurements that could provide the ifs), and work at companies within governments to develop commitments (the thens) based on the evaluations.

  I think this sort of overstates the proportion of effort that went into that kind of work. There was also a lot of work that aimed to develop techniques that reduce or improve understanding of misalignment risk (e.g. Redwood’s stuff).

  METR and Apollo and the broader “evals” agenda became the most popular and highest-prestige thing to work on for people in AI safety.

  IMO both METR and Apollo substantially pivoted away from the strategy you’re describing here at least a year ago.

• Buck 15 Feb 2026 20:01 UTC

  5 points

  5

  in reply to: habryka’s comment on: Eli’s short­form feed

  CDT agents will totally self modify into agents that cooperate in twin prisoners dilemma, but my understanding is that the thing it self modifies into (called “son of CDT”) behaves differently than e.g. the thing EDT agents self modify into.

• Buck 15 Feb 2026 7:09 UTC

  14 points

  −5

  in reply to: Eli Tyre’s comment on: Eli’s short­form feed

  Why wouldn’t an early seed AI reason about the ways that it’s decision theory makes it exploitable, or the ways it’s decision theory which it bars it from cooperation with distant superintelligence (just as the the researchers at SI were doing), find the best solution to those problems, and then modify the decision theory?

  I think that decision theory is probably more like values than empirical beliefs, in that there’s no reason to think that sufficiently intelligent beings will converge to the same decision theory. E.g. I think CDT agents self-modify into having a decision theory that is not the same as what EDT agents self-modify into.

  (Of course, like with values, it might be the case that you can make AIs that are “decision-theoretically corrigible”: these AIs should try to not take actions that rely on decision theories that humans might not endorse on reflection, and they should try to help humans sort out their decision theory problems. I don’t have an opinion on whether this strategy is more or less promising for decision theories than for values.)

  (Aside from decision theory and values, the main important thing that I think might be “subjective” is something like your choice over the universal prior.)

• Buck 15 Feb 2026 7:04 UTC

  4 points

  0

  in reply to: habryka’s comment on: Eli’s short­form feed

  I think this is a good argument for understanding basic decision theory points, but I don’t think it leads to you needing to develop any fancier decision theory—arguing about what decision theory AIs will use just requires thinking about descriptive facts about decision theories, rather than coming up with decision theories that work well in limits that aren’t important for the most important kinds of AI futurism (including the AI futurism questions I think you’re talking about here).

• Buck 1 Feb 2026 19:19 UTC

  9 points

  0

  in reply to: Yair Halberstadt’s comment on: On ‘In­vent­ing Tem­per­a­ture’ and the re­al­ness of properties

  Note that temperature does not have this property! Objects generically have different heat capacities at different temperatures.

• Buck 1 Feb 2026 3:12 UTC

  9 points

  3

  on: Buck’s Shortform

  A question that I think would be a fun AI safety interview question, except that it would give too much of an advantage to people who happened to have thought about the relevant topics before: Explain the differences in risks posed by paperclip maximizers and paperclip minimizers.

• Buck 29 Jan 2026 16:38 UTC

  66 points

  1

  on: Buck’s Shortform

  @ryan_greenblatt and I are going to record another podcast together. We’d love to hear topics that you’d like us to discuss. (The questions people proposed last time are here, for reference.)

• Buck 27 Jan 2026 20:05 UTC

  LW: 7 AF: 5

  0

  AF

  in reply to: Wei Dai’s comment on: Wei Dai’s Shortform

  (I haven’t spelled out what I think about decision theory publicly, so presumably this won’t be completely informative to you. A quick summary is that I think that questions related to decision theory and anthropics are very important for how the future goes, and relevant to some aspects of the earliest risks from misaligned power-seeking AI.)

  My impression is that people at AI companies who have similar opinions to me about risk from misalignment tend to also have pretty similar opinions on decision theory etc. AI company staff with more different opinions tend to have not thought much about decision theory etc.

• Buck 27 Jan 2026 17:27 UTC

  LW: 18 AF: 9

  6

  AF

  in reply to: Wei Dai’s comment on: Wei Dai’s Shortform

  Fwiw I’m not sure this is right; I think that a lot of questions about decision theory become pretty obvious once you start thinking about digital minds and simulations. And my guess is that a lot of FDT-like ideas would have become popular among people like the ones I work with, once people were thinking about those questions.

• Buck 15 Jan 2026 19:47 UTC

  9 points

  0

  on: Reflec­tions on TA-ing Har­vard’s first AI safety course

  I really enjoyed speaking! The students had lots of great questions.

• Buck 14 Jan 2026 16:34 UTC

  23 points

  0

  on: Back­yard cat fight shows Schel­ling points pre­ex­ist language

  if you like to think about this kind of thing, I recommend actually reading Schelling’s “the strategy of conflict”; I thought it had a bunch of interesting points that haven’t made their way into the water supply.

• Buck 13 Jan 2026 5:35 UTC

  4 points

  0

  in reply to: MichaelLowe’s comment on: David Ma­tolcsi’s Shortform

  I think I disagree—doing research like this (especially several such projects) is really helpful for getting hired!

The inau­gu­ral Red­wood Re­search podcast

• Buck 3 Jan 2026 4:27 UTC

  4 points

  2

  in reply to: Bronson Schoen’s comment on: Over­whelming Superintelligence

  Yeah, I think control is unlikely to work for galaxy brained superintelligences. It’s unclear how superintelligent they have to be before control is totally unworkable.

• Buck 3 Jan 2026 4:26 UTC

  4 points

  2

  in reply to: Raemon’s comment on: Over­whelming Superintelligence

  Were you suggesting something other than “remove the parentheses?” Or did it seem like I was thinking about it in a confused way? Not sure which direction you thought the mistake was in.

  I think that it is worth conceptually distinguishing AIs that are uncontrollable from AIs that are able to build uncontrollable AIs, because the way you should handle those two kinds of AI are importantly different.

• Buck 3 Jan 2026 4:25 UTC

  6 points

  2

  in reply to: Raemon’s comment on: Over­whelming Superintelligence

  I think this more feature-than-bug – the problem is that it’s overwhelming. There are multiple ways to be overwhelming, what we want to avoid is a situation where an overwhelming, unfriendly AI exists. One way is not build AI of a given power level. The other is to increase the robustness of civilization. (I agree the term is fuzzy, but I think realistically the territory is fuzzy).

  When you’re thinking about how to mitigate the risks, it really matters which of these we’re talking about. I think there is some level of AI capability at which it’s basically hopeless to control the AIs; this is what I use “galaxy-brained superintelligence” to refer to. If you just want to talk about AIs that pose substantial risk of takeover, you probably shouldn’t use the word superintelligence in there, because they don’t obviously have to be superintelligences to pose takeover risk. (And it’s weird to use “overwhelmingly” as an adverb that modifies “superintelligent”, because the overwhelmingness isn’t about the level of intelligence, it’s about that and also the world. You could say “overwhelming, superintelligent AI” if you want to talk specifically about AIs that are overwhelming and also superintelligent, but that’s normally not what we want to talk about.)

• Buck 2 Jan 2026 19:51 UTC

  4 points

  0

  in reply to: Raemon’s comment on: Over­whelming Superintelligence

  I don’t really understand what you’re saying. I think it’s very likely that [ETA: non-galaxy-brained] superintelligent AIs will be able to build galaxy-brained superintelligences within months to years if they are given (or can steal) the resources needed to produce them. I don’t think it’s obvious that they can do this with extremely limited resources.

• Buck 2 Jan 2026 19:50 UTC

  10 points

  2

  on: Over­whelming Superintelligence

  I agree that it’s useful to have the concept of “Superintelligence that is so qualitatively intelligent that it’s very hard for us to be confident about what it will or won’t be able to accomplish, even given lots of constraints and limited resources.” I usually use “galaxy-brained superintelligence” for this in conversation, but obviously that’s a kind of dumb term. Maybe “massively qualitatively superintelligent” works? Bostrom uses “quality superintelligence”.

  OpenPhil talked about the concept of “transformative AI” specifically because they were trying to talk about a broader class of AIs (though galaxy-brained superintelligence was a core part of their concern).

  I don’t love “overwhelmingly superintelligent” because AIs don’t necessarily have to be qualitatively smarter than humanity to overwhelm it—whether we are overwhelmed by AIs that are “superintelligent” (in the weak sense that they’re qualitatively more intelligent than any human) IMO is affected by the quality of takeover countermeasures in place.

  The type of AI I’m most directly worried about is “overwhelmingly superhuman compared to humanity.” (And, AIs that might quickly bootstrap to become overwhelmingly superhuman).

  I think it’s a mistake to just mention that second thing as a parenthetical. There’s a huge difference between AIs that are already galaxy-brained superintelligences and AIs that could quickly build galaxy-brained superintelligences or modify themselves into galaxy-brained superintelligences—we should try to prevent the former category of AIs from building galaxy-brained superintelligences in ways we don’t approve of.