Dan Braun

• Dan Braun 30 May 2022 10:26 UTC

  12 points

  7

  in reply to: RHollerith’s comment on: Will work­ing here ad­vance AGI? Help us not de­stroy the world!

  I have a different intuition here; I would much prefer the alignment team at e.g. DeepMind to be working at DeepMind as opposed to doing their work for some “alignment-only” outfit. My guess is that there is a non-negligible influence that an alignment team can have on a capabilities org in the form of:

  • The alignment team interacting with other staff either casually in the office or by e.g. running internal workshops open to all staff (like DeepMind apparently do)

  • The org consulting with the alignment team (e.g. before releasing models or starting dangerous projects)

  • Staff working on raw capabilities having somewhere easy to go if they want to shift to alignment work

  I think the above benefits likely outweigh the impact of the influence in the other direction (such as the value drift from having economic or social incentives linked to capabilities work)

• Dan Braun 4 Nov 2023 10:57 UTC

  LW: 7 AF: 4

  3

  AF

  on: Un­trusted smart mod­els and trusted dumb models

  Nice post.

  Pushing back a little on this part of the appendix:

  Also, unlike many other capabilities which we might want to evaluate, we don’t need to worry about the possibility that even though the model can’t do this unassisted, it can do it with improved scaffolding—the central deceptive alignment threat model requires the model to think its strategy through in a forward pass, and so if our model can’t be fine-tuned to answer these questions, we’re probably safe.

  I’m a bit concerned about people assuming this is true for models going forward. A sufficiently intelligent RL-trained model can learn to distribute its planning across multiple forward passes. I think your claim is true for models trained purely on next-token prediction, and for GPT4-level models which, even though they have an RL component in their training, their outputs are all human-understandable (and incorporated into the oversight process).

  But even 12 months from now I’m unsure how confident you could be in this claim for frontier models. Hopefully, labs are dissuaded from producing models which can use uninterpretable scratch pads given how much more dangerous they would be and harder to evaluate.

• Dan Braun 26 Apr 2024 8:44 UTC

  6 points

  2

  on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  This is neat, nice work!

  I’m finding it quite hard to get a sense at what the actual Loss Recovered numbers you report are, and to compare them concretely to other work. If possible, it’d be very helpful if you shared:

  1. What the zero ablations CE scores are for each model and SAE position. (I assume it’s much worse for the MLP and attention outputs than the residual stream?)

  2. What the baseline CE scores are for each model.

• Dan Braun 26 Sep 2023 7:03 UTC

  5 points

  2

  in reply to: aogara’s comment on: Un­der­stand­ing strate­gic de­cep­tion and de­cep­tive alignment

  (These are my own takes, the other authors may disagree)
  
  We briefly address a case that can be viewed as “strategic sycophancy” case in Appendix B in the blog post, which is described similarly to your example. We indeed classify it as an instance of Deceptive Alignment.
  As you mention, this case does have some differences with ideas commonly associated with Deceptive Alignment, notably the difference in behaviour between oversight and non-oversight. But it does share two important commonalities:

  • The model is pursuing a goal that its designers do not want.

  • The model strategically deceives the user (and designer) to further a goal.

  Detecting instances of models that share these properties will likely involve using many of the tools and techniques that would be applied to more canonical forms of deceptive alignment (e.g. evals that attempt to alter/​hamstring a model and measure behaviour in a plethora of settings, interpretability).

  Though, as you mention, preventing/​fixing models that exhibit these properties may involve different solutions, and somewhat crude changes to the training signal may be sufficient for preventing strategic sycophancy (though by doing so you might end up with strategic deception towards some other Misaligned goal).

  I agree that deception which is not strategic or intentional could be important to prevent. However,

  1. I expect the failure cases in these scenarios to manifest earlier, making them easier to fix and likely less catastrophic than cases that are strategic and intentional.

  2. Having a definition of Deceptive Alignment that captured every dangerous behaviour related to deception wouldn’t be very useful. We can use “deception” on its own to refer to this set of cases, and reserve terms like Strategic Deception and Deceptive Alignment for subclasses of deception, ideally subclasses that meaningfully narrow the solution space for detection and prevention.

• Dan Braun 12 Mar 2023 10:33 UTC

  LW: 4 AF: 2

  1

  AF

  on: Un­der­stand­ing and con­trol­ling a maze-solv­ing policy network

  Nice project and writeup. I particularly liked the walkthrough of thought processes throughout the project

  Decision square’s Euclidean distance to the top-right $5\times 5$ corner, positive ($+1.326$).

  We are confused and don’t fully understand which logical interactions produce this positive regression coefficient.

  I’d be weary about interpreting the regression coefficients of features that are correlated (see Multicollinearity). Even the sign may be misleading.

  It might be worth making a cross-correlation plot of the features. This won’t give you a new coefficients to put faith in, but it might help you decide how much to trust the ones you have. It can also be useful looking at how unstable the coefficients are during training (or e.g. when trained on a different dataset).

  What links here?

  • Be­havi­oural statis­tics for a maze-solv­ing agent by peligrietzer (20 Apr 2023 22:26 UTC; 44 points)

• Dan Braun 22 Apr 2023 7:56 UTC

  3 points

  0

  in reply to: TurnTrout’s comment on: Un­der­stand­ing and con­trol­ling a maze-solv­ing policy network

  Thanks for sharing that analysis, it is indeed reassuring!

• Dan Braun 13 Oct 2022 12:53 UTC

  2 points

  1

  in reply to: Nora Belrose’s comment on: In­ter­pret­ing Neu­ral Net­works through the Poly­tope Lens

  Hi Nora. We used rapidsai’s cuml which has GPU compatibility. Beware, the only “metric” available is “euclidean”, despite what the docs say (issue).

• Dan Braun 4 Feb 2023 11:24 UTC

  1 point

  1

  in reply to: Raemon’s comment on: Small Talk is Good, Actually

  Bad link

• Dan Braun 18 Jul 2022 5:16 UTC

  1 point

  0

  in reply to: shminux’s comment on: All AGI safety ques­tions wel­come (es­pe­cially ba­sic ones) [monthly thread]

  I think the risk level becomes clearer when stepping back from stories of how pursuing specific utility functions lead to humanity’s demise. An AGI will have many powerful levers on the world at its disposal. Very few combinations of lever pulls result in a good outcome for humans.
  
  From the perspective of ants in an anthill, the actual utility function(s) of the humans is of minor relevance; the ants will be destroyed by a nuclear bomb in much the same way as they will be destroyed by a new construction site or a group of mischievous kids playing around.

  (I think your Fermi AGI paradox is a good point, I don’t quite know how to factor that into my AGI risk assessment.)

• Dan Braun 29 Mar 2022 7:08 UTC

  1 point

  on: A Longlist of The­o­ries of Im­pact for Interpretability

  Nice list!
  
  Conditioned on the future containing AIs that are capable of suffering in a morally relevant way, interpretability work may also help identify and even reduce this suffering (and/​or increase pleasure and happiness). While this may not directly reduce x-risk, it is a motivator for people taken in by arguments on s-risks from sentient AIs to work on/​advocate for interpretability research.