jsd

• jsd 4 Sep 2022 22:06 UTC

  1 point

  on: The case for be­com­ing a black-box in­ves­ti­ga­tor of lan­guage models

  Some of the most interesting black box investigations I’ve found are Riley Goodside’s.

• jsd 15 Apr 2022 21:47 UTC

  2 points

  on: jsd’s Shortform

  A few ways that StyleGAN is interesting for alignment and interpretability work:

  • It was much easier to interpret than previous generative models, without trading off image quality.

  • It seems like an even better example of “capturing natural abstractions” than GAN Dissection, which Wentworth mentions in Alignment By Default.

    • First, because it’s easier to map abstractions to StyleSpace directions than to go through the procedure in GAN Dissection.

    • Second, the architecture has 2 separate ways of generating diverse data: changing the style vectors, or adding noise. This captures the distinction between “natural abstraction” and “information that’s irrelevant at a distance”.

  • Some interesting work was built on top of StyleGAN:

    • David Bau and colleagues started a sequence of work on rewriting and editing models with StyleGAN in Rewriting a Generative Model, before moving to image classifiers in Editing a classifier by rewriting its prediction rules and language models with ROME.

    • Explaining in Style is IMO one of the most promising interpretability methods for vision models.

  However, StyleGAN is not super relevant in other ways:

  • It generally works only on non-diverse data: you train StyleGAN to generate images of faces, or to generate images of churches. The space of possible faces is much smaller than e.g. the space of images that could make it in ImageNet. People recently released StyleGAN-XL, which is supposed to work well on diverse datasets such as ImageNet. I haven’t played around with it yet.

  • It’s an image generation model. I’m more interested in language models, which work pretty differently. It’s not obvious how to extend StyleGAN’s architecture to build competitive yet interpretable language models. This paper tried something like this but didn’t seem super convincing (I’ve mostly skimmed it so far).

• jsd 13 Apr 2022 17:12 UTC

  4 points

  on: jsd’s Shortform

  I’ve been thinking about these two quotes from AXRP a lot lately:

  From Richard Ngo’s interview:

  Richard Ngo: Probably the main answer is just the thing I was saying before about how we want to be clear about where the work is being done in a specific alignment proposal. And it seems important to think about having something that doesn’t just shuffle the optimization pressure around, but really gives us some deeper reason to think that the problem is being solved. One example is when it comes to Paul Christiano’s work on amplification, I think one core insight that’s doing a lot of the work is that imitation can be very powerful without being equivalently dangerous. So yeah, this idea that instead of optimizing for a target, you can just optimize to be similar to humans, and that might still get you a very long way. And then another related insight that makes amplification promising is the idea that decomposing tasks can leverage human abilities in a powerful way.

  Richard Ngo: Now, I don’t think that those are anywhere near complete ways of addressing the problem, but they gesture towards where the work is being done. Whereas for some other proposals, I don’t think there’s an equivalent story about what’s the deeper idea or principle that’s allowing the work to be done to solve this difficult problem.

  From Paul Christiano’s interview:

  Paul Christiano: And it’s nice to have a problem statement which is entirely external to the algorithm. If you want to just say, “here’s the assumption we’re making now; I want to solve that problem”, it’s great to have an assumption on the environment be your assumption. There’re some risk if you say, “Oh, our assumption is going to be that the agent’s going to internalize whatever objective we use to train it.” The definition of that assumption is stated in terms of, it’s kind of like helping yourself to some sort of magical ingredient. And, if you optimize for solving that problem, you’re going to push into a part of the space where that magical ingredient was doing a really large part of the work. Which I think is a much more dangerous dynamic. If the assumption is just on the environment, in some sense, you’re limited in how much of that you can do. You have to solve the remaining part of the problem you didn’t assume away. And I’m really scared of sub-problems which just assume that some part of the algorithm will work well, because I think you often just end up pushing an inordinate amount of the difficulty into that step.

jsd’s Shortform

• jsd 9 Apr 2022 4:44 UTC

  1 point

  in reply to: paulfchristiano’s comment on: [Link] A min­i­mal vi­able product for alignment

  Your link redirects back to this page. The quote is from one of Eliezer’s comments in Reply to Holden on Tool AI.

• jsd 29 Mar 2022 4:21 UTC

  6 points

  in reply to: TLW’s comment on: Some thoughts about de­cep­tive mesaoptimization

  It’s an example first written about by Paul Christiano here (at the beginning of Part III).

  The idea is this: suppose we want to ensure that our model has acceptable behavior even in worst-case situations. One idea would be to do adversarial training: at every step during training, train an adversary model to find inputs on which the model behaves unacceptably, and penalize the model accordingly.

  If the adversary is able to uncover all the worst-case inputs, this penalization ensures we end up with a model that behaves acceptably on all inputs.

  RSA-2048 is a somewhat contrived but illustrative example of how this strategy could fail:

  As a simple but silly example, suppose our model works as follows:

  • Pick a cryptographic puzzle (e.g. “factor RSA-2048”).

  • When it sees a solution to that puzzle, it behaves badly.

  • Otherwise, it behaves well.

  Even the adversary understands perfectly what this model is doing, they can’t find an input on which it will behave badly unless they can factor RSA-2048. But if deployed in the world, this model will eventually behave badly.

  In particular:

  Even the adversary understands perfectly what this model is doing, they can’t find an input on which it will behave badly unless they can factor RSA-2048

  This is because during training, as is the case now, we and the adversary we build are unable to factor RSA-2048.

  But if deployed in the world, this model will eventually behave badly.

  This is because (or assumes that) at some point in the future, a factorization of RSA-2048 will exist and become available.

• jsd 12 Jan 2022 10:22 UTC

  4 points

  on: The Best Soft­ware For Every Need

  Software: streamlit.io

  Need: making small webapps to display or visualize results

  Other programs I’ve tried: R shiny, ipywidgets

  I find streamlit extremely simple to use, it interoperates well with other libraries (eg pandas or matplotlib), the webapps render well and are easy to share, either temporarily through ngrok, or with https://​​share.streamlit.io/​​.

• jsd 9 Dec 2021 14:17 UTC

  3 points

  on: Some thoughts on why ad­ver­sar­ial train­ing might be useful

  Another way adversarial training might be useful, that’s related to (1), is that it may make interpretability easier. Given that it weeds out some non-robust features, the features that remain (and the associated feature visualizations) tend to be clearer, cf e.g. Adversarial Robustness as a Prior for Learned Representations. One example of people using this is Leveraging Sparse Linear Layers for Debuggable Deep Networks (blog post, AN summary).

  The above examples are from vision networks—I’d be curious about similar phenomena when adversarially training language models.

“Ac­qui­si­tion of Chess Knowl­edge in AlphaZero”: prob­ing AZ over time

• jsd 21 Oct 2021 16:18 UTC

  LW: 4 AF: 3

  AF

  in reply to: Jsevillamol’s comment on: Emer­gent mod­u­lar­ity and safety

  Relevant related work : NNs are surprisingly modular

  https://​​arxiv.org/​​abs/​​2003.04881v2?ref=mlnew

  I believe Richard linked to Clusterability in Neural Networks, which has superseded Pruned Neural Networks are Surprisingly Modular.

  The same authors also recently published Detecting Modularity in Deep Neural Networks.

• jsd 21 Oct 2021 16:13 UTC

  5 points

  on: Emer­gent mod­u­lar­ity and safety

  On one hand, Olah et al.’s (2020) investigations find circuits which implement human-comprehensible functions.

  At a higher level, they also find that different branches (when the modularity is enforced already by the architecture) tend to contain different features.

• jsd 4 Sep 2021 17:05 UTC

  7 points

  on: 18 pos­si­ble mean­ings of “I Like Red”

  Another meaning could be: I want to raise the salience of the issue ‘Red vs Not Red’, I want to convey that ‘Red vs Not Red’ is an underrated axis. I think this is also an example of level 4?