Buck

Rogue in­ter­nal de­ploy­ments via ex­ter­nal APIs

• Buck 11 Oct 2025 20:32 UTC

  13 points

  11

  in reply to: davekasten’s comment on: dav­ekas­ten’s Shortform

  I basically agree with Zach that based on public information it seems like it would be really hard for them to be robust to this and it seems implausible that they have justified confidence in such robustness.

  I agree that he doesn’t say the argument in very much depth. Obviously, I think it’d be great if someone made the argument in more detail. I think Zach’s point is a positive contribution even though it isn’t that detailed.

• Buck 11 Oct 2025 3:39 UTC

  5 points

  0

  in reply to: Raemon’s comment on: Rae­mon’s Short­form Feed

  The most common ways that I see comments have errors that I think an LLM could fix are:

  • Typos.

  • Missing a point. Like I often write a comment and fail to realize that someone nearby in the comment tree has already responded to this point. Or I misunderstood the comment I was responding to. It would be helpful to have LLMs note this.

  • Maybe basic fact-checking?

  Maybe you should roll this out for comments before posts.

• Buck 9 Oct 2025 20:36 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: Adam Karvonen’s comment on: The Think­ing Machines Tinker API is good news for AI con­trol and security

  Yeah for sure. A really nice thing about the Tinker API is that it doesn’t allow users to specify arbitrary code to be executed on the machine with weights, which makes security much easier.

• Buck 9 Oct 2025 19:18 UTC

  113 points

  107

  on: Buck’s Shortform

  I hear a lot of scorn for the rationalist style where you caveat every sentence with “I think” or the like. I want to defend that style.

  There is real semantic content to me saying “I think” in a sentence. I don’t say it when I’m stating established fact. I only use it when I’m saying something which is fundamentally speculative. But most of my sentences are fundamentally speculative.

  It feels like people were complaining that I use the future tense a lot. Like, sure, my text uses the future tense more than average, and future tense is indeed somewhat more awkward. But future tense is the established way to talk about the future, which is what I wanted to talk about. It seems pretty weird to switch to present tense just because people don’t like future tense.

• Buck 9 Oct 2025 19:18 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: Adam Karvonen’s comment on: The Think­ing Machines Tinker API is good news for AI con­trol and security

  Yeah, what I’m saying is that even if the computation performed in a hook is trivial, it sucks if that computation has to happen on a different computer than the one doing inference.

• Buck 9 Oct 2025 17:46 UTC

  LW: 6 AF: 3

  0

  AF

  in reply to: ryan_greenblatt’s comment on: The Think­ing Machines Tinker API is good news for AI con­trol and security

  Yeah totally there’s a bunch of stuff like this you could do. The two main issues:

  • Adding methods like this might increase complexity and if you add lots of them they might interact in ways that allow users to violate your security properties.

  • Some natural things you’d want to do for interacting with activations (e.g. applying arbitrary functions to modify activations during a forward pass) would substantially reduce the efficiency and batchability here—the API server would have to block inference while waiting for the user’s computer to compute the change to activations and send it back.

  It would be a slightly good exercise for someone to go through the most important techniques that interact with model internals and see how many of them would have these problems.

• Buck 9 Oct 2025 17:31 UTC

  21 points

  7

  in reply to: Tomás B.’s comment on: Tomás B.’s Shortform

  (For clarity: Open Phil funded those guys in the sense of funding Epoch, where they previously worked and where they probably developed a lot of useful context and connections, but AFAIK hasn’t funded Mechanize.)

The Think­ing Machines Tinker API is good news for AI con­trol and security

• Buck 8 Oct 2025 17:08 UTC

  4 points

  2

  on: Gen­er­al­iza­tion and the Mul­ti­ple Stage Fal­lacy?

  I liked Joe Carlsmith’s discussion of this here.

• Buck 7 Oct 2025 3:20 UTC

  15 points

  2

  on: We won’t get AIs smart enough to solve al­ign­ment but too dumb to rebel

  The argument in this post seems to be:

  AIs smart enough to help with alignment are capable enough that they’ll realize they are misaligned. Therefore, they will not help with alignment.

  When I think about getting misaligned AIs to help with alignment research and other tasks, I’m normally not imagining that the AIs are unaware that they are misaligned. I’m imagining that we can get them to do useful work anyway. See here and here.

  You might be interested in the Redwood Research reading list, which contains lots of analyses of these questions and many others.

• Buck 3 Oct 2025 15:40 UTC

  2 points

  1

  in reply to: Mikhail Samin’s comment on: Rea­sons to sell fron­tier lab equity to donate now rather than later

  As someone who’s worked at MIRI, I disagree regardless of when you are imagining them doing this.

  Conditioned on agreeing with them about AI xrisk stuff and also about high level strategy, I think giving them money now seems better than in the past.

• Buck 2 Oct 2025 15:21 UTC

  2 points

  0

  in reply to: ChristianKl’s comment on: Buck’s Shortform

  Strong upvoted to signal boost, but again note I don’t know what I’m talking about.

• Buck 1 Oct 2025 17:51 UTC

  8 points

  8

  in reply to: Aaron_Scher’s comment on: A Re­ply to MacAskill on “If Any­one Builds It, Every­one Dies”

  Note that most of the compute in consumer laptops is in their GPUs not their CPUs, so comparing H100 flops to laptop CPU flops does not work for establishing the extent to which your policy would affect consumer laptops.

• Buck 30 Sep 2025 4:25 UTC

  5 points

  0

  in reply to: Raemon’s comment on: Buck’s Shortform

  What I hear is that the natsec people judge people for using “treaty” in cases like this. Maybe MIRI looked into it and has better info than me; idk.

• Buck 30 Sep 2025 4:07 UTC

  44 points

  2

  on: Buck’s Shortform

  I hear a lot of discussion of treaties to monitor compute or ban AI or whatever. But the word “treaty” has a specific meaning that often isn’t what people mean. Specifically, treaties are a particular kind of international agreement that (in the US) require Senate approval, and there are a lot of other types of international agreement (e.g. executive agreements that are just made by the president). Treaties are generally more serious and harder to withdraw from.

  As a total non-expert, “treaty” does actually seem to me like the vibe that e.g. our MIRI friends are going for—they want something analogous to major nuclear arms control agreements, which are usually treaties. So to me, “treaty” seems like a good word for the central example of what MIRI wants, but international agreements that they’re very happy about won’t necessarily be treaties.

  I wouldn’t care about this point, except that I’ve heard that national security experts are prickly about the word “treaty” being used when “international agreement” could have been used instead (and if you talk about treaties they might assume you’re using the word ignorantly even if you aren’t). My guess is that you should usually conform to their shibboleth here and say “international agreement” when you don’t specifically want to talk about treaties.

• Buck 29 Sep 2025 1:27 UTC

  12 points

  0

  in reply to: Chris Lakin’s comment on: Rea­sons to sell fron­tier lab equity to donate now rather than later

  For some donation opportunities, e.g. political donations, that would be a crime.

• Buck 28 Sep 2025 22:46 UTC

  41 points

  6

  in reply to: Eliezer Yudkowsky’s comment on: A non-re­view of “If Any­one Builds It, Every­one Dies”

  Where do you think you’ve spelled this argument out best? I’m aware of a lot of places where you’ve made the argument in passing, but I don’t know of anywhere where you say it in depth.

  My response last time (which also wasn’t really in depth; I should maybe try to articulate my position better sometime...) was this:

  I agree that the regime where mistakes don’t kill you isn’t the same as the regime where mistakes do kill you. But it might be similar in the relevant respects. As a trivial example, if you build a machine in America it usually works when you bring it to Australia. I think that arguments at the level of abstraction you’ve given here don’t establish that this is one of the cases where the risk of the generalization failing is high rather than low.

• Buck 28 Sep 2025 22:06 UTC

  9 points

  6

  in reply to: evhub’s comment on: Tomás B.’s Shortform

  The way I’d think about this is:

  Currently, intellectual labor from machine learning researchers costs a lot of compute. A $1M/​year ML researcher costs the same as having 30 or so H100s. At the point where you have AGI, you can probably run the equivalent of one ML researcher with substantially less hardware than that. (I’m amortizing, presumably you’ll be running your models on multiple chips doing inference on multiple requests simultaneously.) This means that some ways to convert intellectual labor into compute efficiency will be cost-effective when they weren’t previously. So I expect that ML will become substantially more labor-intensive and have much more finicky special casing.

• Buck 28 Sep 2025 19:07 UTC

  9 points

  0

  on: Prompt op­ti­miza­tion can en­able AI con­trol research

  Thanks for doing this. This result is helpful. We at Redwood will definitely try out these methods.