Dan Braun

• Dan Braun 29 Aug 2026 18:05 UTC

  2 points

  0

  on: AI Tweets

  If AI goes well I think the world could be extremely good. The bottom 10% living far better than the top 10% do now. This is the promise that gets people working so hard to build something so dangerous.

  Insofar as that last sentence is true, I think it’d be true mostly because it provides cover (to oneself and others) for other motivations like working on cool problems with smart people at high profile companies and getting large compensation.

• Dan Braun 25 Jul 2026 20:57 UTC

  5 points

  0

  on: LLMs are (still) mostly pow­ered by imi­ta­tive learn­ing, not RL

  It might be interesting to look at KL divergence or weight changes between randomly-initialized models versus base models, and between base models versus RLVR? I can’t immediately find literature on that.

  A colleague (Ekdeep Singh) sent me:

  There’s quite a bit of work on this: I did a couple of papers in synthetic scenarios (formal languages or synthetic data from LLMs) to see how base + SFT versus learn the data from scratch differ. My friend Andrew extended the analysis for DPO and RLVR. Old BERT era had papers like this, claiming updates are very sparse during SFT; similar work has been done for RLVR in the modern era now (also see this one for token level change analysis). High level, not much changes in a model via post-training methods, and on pretraining data the model has low KL w.r.t. base model.

  Also arxiv.org/​​pdf/​​2312.04828 (from another colleague).

• Dan Braun 23 Jul 2026 8:25 UTC

  2 points

  2

  in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Per­l­man’s Shortform

  If this AI were embodied, and the answer sheet were sitting on a desk in another room in the same building, I don’t think it’s a big stretch to imagine it physically harming humans to get access.

  That scenario would have landed very differently with the rest of the world. Godspeed to those trying to help people recognise that.

• Dan Braun 11 Jun 2026 22:58 UTC

  2 points

  0

  in reply to: ryan_greenblatt’s comment on: ryan_green­blatt’s Shortform

  I think I agree. Though one (maybe fanciful) hope I have is that a non-superhuman AI that is good at long-term planning and has strong drives will be more effective at preventing unaligned super intelligences from being developed. Even if just for its own sake.

  Also +1 to Cleo Nardo’s point: [rephrased] instilling long-run objectives into an AI may make it more robust cyber thieves/​attackers creating a worldeater from it, or limit the blast radius of accidents.

• Dan Braun 28 May 2026 8:16 UTC

  5 points

  6

  in reply to: Buck’s comment on: Dan Braun’s Shortform

  Oh, I didn’t mean to imply that getting funded at all for ambitious research wasn’t possible or was extremely difficult. The directions I mentioned above are all funded to some extent. Just that e.g. I expect Steven could get 10x+ financial compensation by joining a lab and doing less ambitious AI safety research (or, with less confidence, get more compensation and credit by starting a for-profit or even a non-profit doing less ambitious work). And that people making these sacrifices should be lauded.

• Dan Braun 28 May 2026 0:20 UTC

  9 points

  5

  in reply to: Buck’s comment on: Dan Braun’s Shortform

  There are two (overlapping) groups that I wanted to thank and incentivise which I bundled together in the OP. In hindsight I probably should have separated them out.

  1. Those that pursue research because they think it’s valuable and are foregoing big bucks and external credibility. This could be ambitious or incremental work.

  2. Those that aim to solve fundamental problems that would lead to step/​changes in our ability to control/​guide AI.

  Point 1 is uncontroversial. 2 is cruxy to the extent that people disagree about the expected value safety-wise of ambitious and incremental work.

  My guess is that I think E(ambitious) - E(incremental) is larger than you do. Miscellaneous grab bag of ambitious work that I think is or was high EV:

  - some mech interp (fwiw I’d call work that e.g. applies SAEs to a new problem or tries to minorly improve them incremental research rather than ambitious.)

  - deep learning theory

  - agent foundations

  - Steven Byrnes brain-like AGI stuff

  Then yeah I also claimed that jobs and funding are biased against ambitious research. As you mentioned, the case for jobs is clear. RE funding I think it’s very hard to evaluate ambitious proposals that typically don’t have good short-term milestones. It at least seems like CG are actively trying to overcome that. More varied funding sources would help too.

• Dan Braun 27 May 2026 9:47 UTC

  2 points

  0

  in reply to: Dan Braun’s comment on: Dan Braun’s Shortform

  To be clear, when I said “unfortunately, job and funding incentives are biased against research ambition”, this wasn’t a claim that this is an issue that can and should be easily fixed by AI safety grantmakers. Maybe it could be, I don’t know enough to make a call there. I’d make the same statement about probably any research field. Grantmaking to projects that don’t have short-term milestones, often involving researchers without legible experience, would be very difficult. Same goes for giving jobs to people that do that kind of research.

  While I’m glad people are discussing grantmaking, this was simply an appreciation post for people who have done or are doing this kind of research without 7-figure salaries or much external credibility.

• Dan Braun 27 May 2026 0:56 UTC

  2 points

  2

  in reply to: Jacob Pfau’s comment on: Dan Braun’s Shortform

  My first sentence was very much a finger to the wind. Classifying people as AI safety researchers would be a post in itself.

  Agreed that there‘s high value to narrowly-scoped research. I encourage people to do it, especially new researchers. But people should be mindful that they’re doing it at all and why they’re doing it.

• Dan Braun 26 May 2026 17:03 UTC

  10 points

  2

  in reply to: leogao’s comment on: Dan Braun’s Shortform

  I’m not very clued in to the current grantmaker space. But I know that top researchers in interp are queried far less about funding proposals than what they can handle. I expect it’s the same for other fields.

Dan Braun’s Shortform

• Dan Braun 25 May 2026 15:38 UTC

  152 points

  100

  on: Dan Braun’s Shortform

  I wish the median AI safety researcher were much more ambitious with the problems they choose to tackle. Unfortunately, job and funding incentives are biased against research ambition.

  I’d thus like to celebrate the people who have taken risks to pursue ambitious AI safety research directions that have not panned out (or have yet to pan out!). These people will not have received riches and accolades from the field, or even their close peers.

  For much of the work that falls in this category, it might be obvious to many others at the time that it isn’t going to lead anywhere. Unless their efforts were likely to be actively harmful, I still want to celebrate these people for their courage, and for pushing against the incentive gradient.

  I was planning on making a long list of work that I thought fell into this category, but quickly felt uncomfortable including and excluding people’s work without more thorough analysis that I didn’t think was worth it. Maybe some day I will.

  If you think that you’ve done work that falls into this category, thank you for your +EV. Humanity is grateful for your efforts.

[Linkpost] In­ter­pret­ing Lan­guage Model Parameters

• Dan Braun 8 Jan 2026 18:16 UTC

  2 points

  0

  on: Two Aspects of Si­tu­a­tional Aware­ness: World Model­ling & In­dex­i­cal Information

  I don’t think this raises a challenge to physicalism.

  If physicalism were true, or even if there were non-physical things but they didn’t alter the determinism of the physical world, then the notion of an “agent” needs a lot of care. It can easily give the mistaken impression that there is something non-physical in entities that can change the physical world.

  A perfect world model would be able to predict the responses of any neuron in any location in the universe to any inputs (leaving aside true randomness). It doesn’t matter whether the entity in question has a conscious experience that it is one of those entities, nothing would change.

  but you don’t know where YOU are in space-time

  So I’d argue that this question is irrelevant if physicalism is true, because the AI having a phenomenal conscious experience of “I am this entity” cannot affect the physical world. If we’re not talking about phenomenal consciousness, then it’s just regular physical world modeling.

• Dan Braun 10 Dec 2025 16:48 UTC

  4 points

  7

  in reply to: Mo Putera’s comment on: Mo Putera’s Shortform

  That’s the trap: in software, effort is easy to generate, activity is easy to justify, and impact is surprisingly easy to avoid.

  Yyep. And it’s much much worse for research.

• Dan Braun 6 Dec 2025 9:37 UTC

  6 points

  0

  in reply to: Roman Malov’s comment on: Ro­man Malov’s Shortform

  I know your comment isn’t an earnest attempt to convince people, but fwiw:

  For years, AI x-risk people have warned us that a huge danger comes with AI capable of RSI

  I think this argument is more likely to have the opposite effect than intended when used on the types of people pushing on RSI. I think your final paragraph would be much more effective.

• Dan Braun 11 Oct 2025 17:38 UTC

  16 points

  2

  in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Per­l­man’s Shortform

  The ASL-3 security standard states in 4.2.4 that “third-party environments”, which surely includes compute providers, are in scope (and on their minds) for the standards they laid out:

  Third-party environments: Document how all relevant models will meet the criteria above, even if they are deployed in a third-party partner’s environment that may have a different set of safeguards.

• Dan Braun 22 Aug 2025 17:32 UTC

  14 points

  2

  on: I un­der­es­ti­mated safety re­search speedups from safe AI

  UPDATE

  When writing this post, I think I was biased by the specific work I’d been doing for the 6-12 months prior, and generalised that too far.

  I think I stand by the claim that tooling alone could speedup the research I was working on by 3-5x. But even on the same agenda now, the work is far less amenable to major speedups from tooling. Now, work on the agenda is far less “implement several minor algorithmic variants, run hyperparameter sweeps which take < 1 hour, evaluate a set of somewhat concrete metrics, repeat”, and more “think deeply about which variants make the most sense, run >3 hours jobs/​sweeps, evaluate the more murky metrics”.

  The main change was switching our experiments from toy models to real models, which massively loosened the iteration loop due to increased training time and less clear evaluations.

  For the current state, I think 6-12 months of tooling progress might give 1.5-2x speedup from the baseline 1 year ago.

  I still believe that I underestimated safety research speedups overall, but not by as much as I thought 2 months ago.

  What links here?

  • I un­der­es­ti­mated safety re­search speedups from safe AI by Dan Braun (29 Jun 2025 13:29 UTC; 38 points)

I un­der­es­ti­mated safety re­search speedups from safe AI

[Paper] Stochas­tic Pa­ram­e­ter Decomposition

• Dan Braun 24 Jun 2025 14:17 UTC

  3 points

  2

  on: Com­pressed Com­pu­ta­tion is (prob­a­bly) not Com­pu­ta­tion in Superposition

  I think this is a fun and (initially) counterintuitive result. I’ll try to frame things as it works in my head, it might help people understand the weirdness.

  The task of the residual MLP (labelled CC Model here) is to solve y = x + ReLU(x). Consider the problem from the MLP’s perspective. You might think that the problem for the MLP is to just learn how to compute ReLU(x) for 100 input features with only 50 neurons. But given that we have this random $W_E$ matrix, the task is actually more complicated. Not only does the MLP have to compute ReLU(x), it also has to make up for the mess caused by $W_E{W}_E^T$ not being an identity.

  But it turns out that making up for this mess actually makes the problem easier!