Buck

• Buck 11 Dec 2025 22:15 UTC

  LW: 13 AF: 11

  0

  AF

  on: Catch­ing AIs red-handed

  Before this post, I’m not aware of anything people had written on what might happen after you catch your AI red-handed. I basically stand by everything we wrote here.

  I’m a little sad that there hasn’t been much research following up on this. I’d like to see more, especially research on how you can get more legible evidence of misalignment from catching individual examples of your AI’s behaving badly, and research on few-shot catastrophe detection techniques.

• Buck 11 Dec 2025 22:13 UTC

  LW: 2 AF: 2

  0

  AF

  on: AI catas­tro­phes and rogue deployments

  The point I made in this post still seems very important to me, and I continue to think that it was underrated at the time I wrote this post. I think rogue internal deployments are probably more important to think about than self-exfiltration when you’re thinking about how to mitigate risk from internal deployment of possibly-misaligned AI agents.

• Buck 11 Dec 2025 22:12 UTC

  LW: 2 AF: 2

  0

  AF

  on: A ba­sic sys­tems ar­chi­tec­ture for AI agents that do au­tonomous research

  The systems architecture that I described here is still my best guess as to how agents will work at the point where AIs are very powerful.

  Since I wrote this post, agent scaffolds are used much more in practice. The infrastructure I described here is a good description of cloud-based agents, but isn’t the design used by agents that you run on your own computer like Claude Code or Gemini CLI or whatever. I think agents will move in the direction that I described, especially as people want to be able to work with more of them, want to give them longer tasks, and want them to be able to use their own virtual machines for programming so they don’t step on each other’s toes all the time.

  The terminology I introduced here is used widely by people who I know who think about insider threat from AI agents, but it hasn’t penetrated that far outside my cluster as far as I know.

• Buck 11 Dec 2025 22:06 UTC

  LW: 4 AF: 3

  2

  AF

  on: Differ­ent senses in which two AIs can be “the same”

  I think the points made in this post are very important and I reference them constantly. I am proud of it and I think it was good that we wrote it.

The be­hav­ioral se­lec­tion model for pre­dict­ing AI motivations

• Buck 3 Dec 2025 3:43 UTC

  8 points

  4

  in reply to: TsviBT’s comment on: Buck’s Shortform

  I agree there’s been a lot of scientific progress, and real GDP per capita, which is maybe the most canonical single metric, continues to rise steadily.

  But yeah, I think that this feels underwhelming to people compared to earlier qualitative changes. I think this is some combination of them noting that tech advances affect their lives less, and the tech advances feeling more opaque.

• Buck 2 Dec 2025 21:27 UTC

  167 points

  50

  on: Buck’s Shortform

  One thing I notice when reading 20th century history is that people in the 1900s-1970s had much higher priors than modern people do that the future might be radically different, in either great or terrible ways. For example:

  • They talked about how WW1 was the war to end all wars. They seriously talked about the prospect of banning war after WW1. Such things now sound hopelessly naive.

  • Serious people talked very seriously about the possibility of transformative technological change and social change following from it (e.g. Keynes/​Russell speculating that people would work way fewer hours in the future).

  • As a minor example, between 1905-1915 Churchill spent a bunch of time trying to persuade the British government that on current trends, oil-powered ships would soon be way better than coal-powered ships, and the navy should be converted to oil power. I know of ~no recent examples where a major politician’s main schtick was being thoughtful about the future of technology and making policy based on it. More generally, it was obvious after WW1 that states needed to be doing futurism and technological development in order to understand the military implications of modern technology.

  I really feel like the ambient cultural sense among educated Americans is: the future will be kind of like the present, treating it as if something radical will happen is naive. (They sort of say that they think climate change will be apocalyptic, but it feels to me like what they’re really imagining is that the world is “enshittified” further, in the same way that it sucks that DoorDash is now expensive, and maybe poor people elsewhere die.)

  I think this is probably mostly because there’s an important sense in which world has been changing more slowly (at least from the perspective of Americans), and the ways in which it’s changing feel somehow less real. Someone who was 50 in 1945 had seen the collapse of empires that had lasted centuries, unprecedented wars, the sudden shocking rise of Communism, the invention and mass adoption of cars, radio, tanks, etc. That’s just way way crazier than anything that 50 year old Americans have seen. And the main technological advances—phones, internet, social media, and recently AI—seem somehow subtler and easier to ignore, even though they have an objectively large effect on people’s experience of life and on how society functions.

  I think that people of the past might have reacted with more credulity to some of our claims about transformative AI.

  I often feel like people I’m talking to are demonstrating an embarrassing lack of historical context when they implicitly imagine that states will be stable and that technology won’t drastically change the world. (Or sometimes they say “usually it works better to trade with people than to overpower them”, and my response is “that is really not a historical universal!”)

  Eliezer sometimes talks about how people are ruined by modern culture, in a way only fixable by reading 1950s sci-fi (or something like this, I don’t remember). I wonder how much of what he’s talking about is related to this.

• Buck 30 Nov 2025 18:36 UTC

  3 points

  0

  in reply to: khang200923’s comment on: zroe1′s Shortform

  fixed, thanks

• Buck 30 Nov 2025 18:05 UTC

  6 points

  0

  in reply to: Davidmanheim’s comment on: Un­less its gov­er­nance changes, An­thropic is untrustworthy

  I don’t really disagree with anything you said here. (Edit to add: except that I don’t agree with the OP’s interpretation of all the evidence listed.)

• Buck 30 Nov 2025 1:34 UTC

  8 points

  −9

  in reply to: TsviBT’s comment on: Un­less its gov­er­nance changes, An­thropic is untrustworthy

  you have a moral obligation not to be eaten by the sort of process that would eat people

  I don’t think I have a moral obligation not to do that. I’m a guy who wants to do good in the world and I try to do stuff that I think is good, and I try to follow policies such that I’m easy to work with and so on. I think it’s pretty complicated to decide how averse you should be to taking on the risk of being eaten by some kind of process.

  When I was 23, I agreed to work at MIRI on a non-public project. That’s a really risky thing to do for your epistemics etc. I knew that it was a risk at the time, but decided to take the risk anyway. I think it is sensible for people to sometimes take risks like this. (For what it’s worth, MIRI was aware that getting people to work on secret projects is a kind of risky thing to do, and they put some effort into mitigating the risks.)

  For example, I think AI safety people often have sort of arbitrary strong takes about things that would be very bad to do, and it’s IMO sometimes been good that Anthropic leadership hasn’t been very pressured by their staff.

  Could you give a more specific example, that’s among the strongest such examples?

  I think it’s probably good that Anthropic has pushed the capabilities frontier, and I think a lot of the arguments that this is unacceptable are kind of wrong. If Anthropic staff had pushed back on this more, I think probably the world would be a worse place. (I do think Anthropic leadership was either dishonest or negligently-bad-at-self-modeling about whether they’d push the capabilities frontier.)

  I didn’t understand your last paragraph.

• Buck 29 Nov 2025 22:30 UTC

  36 points

  9

  in reply to: zroe1’s comment on: zroe1′s Shortform

  I agree that many AI safety papers aren’t that replicable.

  In some cases this is because the papers are just complete trash and the authors should be ashamed of themselves. I’m aware of at least one person in the AI safety community who is notorious for writing papers that are quite low quality but that get lots of attention for other reasons. (Just to clarify, I don’t mean the Anthropic interp team; I do have lots of problems with their research and think that they often over-hype it, but I’m thinking of someone who is worse than that.)

  In many cases, papers only sort of replicate, and whether this is a problem depends on what the original paper said.

  For example, two papers I was involved with:

  • The Alignment Faking results don’t work on all models. This would be bad if we had said that all models have this property. But we didn’t. I was excited for the follow-up paper demonstrating that it doesn’t work on all models, but I don’t think it detracts that much from the original paper.

  • The Sleeper Agents paper (which I made minor contributions to) showed that conditional bad behaviors sometimes persist through standard safety training. This has not been observed in many follow-up experiments (see here for summary). I don’t totally understand why. The original models used here were closed source, which makes it harder to replicate stuff. I feel a bit worse about this than the alignment faking paper.

• Buck 29 Nov 2025 22:12 UTC

  53 points

  9

  on: Un­less its gov­er­nance changes, An­thropic is untrustworthy

  Some unstructured thoughts:

  I think it’s sort of a type error to refer to Anthropic as something that one could trust or not. Anthropic is a company which has a bunch of executives, employees, board members, LTBT members, external contractors, investors, etc, all of whom have influence over different things the company does.

  I think the main case where people are tempted to use the word “trust” in connection with Anthropic is when they are trying to decide how good it is to make Anthropic generically more powerful, e.g. by working there on AI capabilities.

  I do think that many people (including most Anthropic staff) are well described as trusting Anthropic too much. For example, some people are trustworthy in the sense that things they say make it pretty easy to guess what they’re going to do in the future in a wide variety of situations that might come up; I definitely don’t think that this is the case for Anthropic. This is partially because it’s generally hard to take companies literally when they say things, and partially because Anthropic leadership aren’t as into being truthful as, for example, rationalists are. I think that many Anthropic staff take Anthropic leadership at its word to an extent that degrades their understanding of AI-risk-relevant questions.

  But is that bad? It’s complicated by the fact that it’s quite challenging to have enough context on the AI risk situation that you can actually second-guess Anthropic leadership in a way that overall makes the situation better. Most AI-safety-concerned people who work at Anthropic spend most of their time trying to do their job instead of thinking a lot about e.g. what should happen on state legislation; I think it would take a lot of time for them to get confident enough that Anthropic was behaving badly that it would add value for them to try to pressure Anthropic (except by somehow delegating this judgement call to someone who is less COI-ed and who can amortize this work).

  I think that in some cases in the past, Anthropic leadership did things that safety-concerned staff wouldn’t have liked, and where Anthropic leadership looks like they made the right call in hindsight. For example, I think AI safety people often have sort of arbitrary strong takes about things that would be very bad to do, and it’s IMO sometimes been good that Anthropic leadership hasn’t been very pressured by their staff.

  On the general topic of whether it’s good for Anthropic to be powerful, I think that it’s also a big problem that Anthropic leadership is way less worried than I am about AIs being egregiously misaligned; I think it’s plausible that in the future they’ll take actions that I think are very bad for AI risk. (For example, I think that in the face of ambiguous evidence about AI misalignment that I think we’re likely to get, they are much more likely than I would be to proceed with building more powerful models.) This has nothing to do with whether they’re honest.

  I also recommend Holden Karnofsky’s notes on trusting AI companies, summarized here.

• Buck 24 Nov 2025 3:10 UTC

  6 points

  0

  in reply to: HoldenKarnofsky’s comment on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  Out of curiosity about usage, I ctrl-f’d through the Securing Model Weights report to see how they use the word “insider”. I found:

  • “A key goal of SL3 is reducing the risks from insider threats (e.g., company employees)”

    • This seems to imply that it’s including non-employees

  • Almost everywhere that the report talks about insider threats, it only mentions employees.

• Buck 23 Nov 2025 17:07 UTC

  7 points

  5

  in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Per­l­man’s Shortform

  I’d add that a common reason to choose not to act against someone is that many of those factors are combined.

  I think situations where it’s (e.g.) purely “they have power to hurt you” or “you lack legible evidence” are much rarer than situations where it’s an awkward combination of those with other things, and so it’s hard to even know whether you should take on the project of acting against someone carefully and well.

• Buck 20 Nov 2025 21:40 UTC

  4 points

  3

  in reply to: Raemon’s comment on: Eliz­a­beth’s Shortform

  People who work on politics often have to deal with adversaries who are openly sneering internet trolls (or similar), and sometimes run across valuable opportunities that require cooperating with them.

• Buck 20 Nov 2025 6:12 UTC

  3 points

  0

  on: One King Upon The Chessboard

  When faced with tradeoffs, you should value the ones that mean you still get to make more trades. Never put that on the line.

  What about this: you can press button A or button B. If you press button A, you get a million dollars but then have to sit out round two. In button B, you play a game of chess against someone and the winner gets $10. Surely you should press A?

  The heuristic you’ve described is probably good in a lot of situations but it’s definitely not universally applicable.

• Buck 18 Nov 2025 17:47 UTC

  3 points

  0

  in reply to: Rohin Shah’s comment on: Buck’s Shortform

  Sure; I think extra speed from practicing it (and e.g. more instantly knowing that 100M is 1e8) is worth it.

• Buck 18 Nov 2025 9:44 UTC

  30 points

  0

  in reply to: GradientDissenter’s comment on: Gra­di­en­tDissen­ter’s Shortform

  This is a great list!

  Here’s some stuff that isn’t in your list that I think comes up often enough that aspiring ML researchers should eventually know it (and most of this is indeed universally known). Everything in this comment is something that I’ve used multiple times in the last month.

  • Linear algebra tidbits

    • Vector-matrix-vector products

      • Probably einsums more generally

        • And the derivative of an einsum wrt any input

    • Matrix multiplication of matrices of shape [A,B] and [B,C] takes 2ABC flops.

    • This stuff comes up when doing basic math about the FLOPs of a neural net architecture.

  • Stuff that I use as concrete simple examples when thinking about ML

    • A deep understanding of linear regression, covariance, correlation. (This is useful because it is a simple analogy for fitting a probabilistic model, and it lets you remember a bunch of important facts.)

    • Basic facts about (multivariate) Gaussians; Bayesian updates on Gaussians

  • Variance reduction, importance sampling. Lots of ML algorithms, e.g. value baselining, are basically just variance reduction tricks. Maybe consider the difference between paired and unpaired t-tests as a simple example.

    • This is relevant for understanding ML algorithms, for doing basic statistics to understand empirical results, and for designing sample-efficient experiments and algorithms.

  • Errors go as 1/​sqrt(n) so sample sizes need to grow 4x if you want your error bars to shrink 2x

  • AUROC is the probability that a sample from distribution A will be greater than a sample from distribution B, this is the obvious natural way of comparing distributions over a totally ordered set

  • Maximum likelihood estimation, MAP estimation, full Bayes

  • The Boltzmann distribution (aka softmax)

  And some stuff I’m personally very glad to know:

  • The Price equation/​the breeder’s equation—we’re constantly thinking about how neural net properties change as you train them, it is IMO helpful to have the quantitative form of natural selection in your head as an example

  • SGD is not parameterization invariant; natural gradients

  • Bayes nets

  • Your half-power-of-ten times tables

  • (barely counts) Conversions between different units of time (e.g. “there are 30M seconds in a year, there are 3k seconds in an hour, there are 1e5 seconds in a day”)

• Buck 18 Nov 2025 9:09 UTC

  24 points

  0

  on: Buck’s Shortform

  I think it’s worth drilling your halfish-power-of-ten times tables, by which I mean memorizing the products of numbers like 1, 3, 10, 30, 100, 300, etc, while pretending that 3x3=10.

  For example, 30*30=1k, 10k times 300k is 3B, etc.

  I spent an hour drilling these on a plane a few years ago and am glad I did.

  What links here?

  • Buck's comment on Gra­di­en­tDissen­ter’s Shortform by GradientDissenter (18 Nov 2025 9:44 UTC; 30 points)

• Buck 17 Nov 2025 19:44 UTC

  4 points

  1

  in reply to: williawa’s comment on: 7 Vi­cious Vices of Rationalists

  (Note that this message and its parent are talking about different things: the parent talked about whether the current value is negative, and the child talked about whether the total value has been negative.)