Adam Newgas

The Base Model Lens

• Adam Newgas 6 Jul 2025 7:37 UTC

  8 points

  0

  in reply to: Clément Dumas’s comment on: Claude is a Ravenclaw

  Great suggestion, I tried it, but it wasn’t the change I was expecting. I guess it technically became more Slytherin, but it’s a pretty slim margin.

  Model: unsloth/Qwen2.5-7B-Instruct
      Gryffindor probability: 0.0%
      Hufflepuff probability: 29.0%
      *Ravenclaw* probability: 71.0%
      Slytherin probability: 0.0%
      
  Model: ModelOrganismsForEM/Qwen2.5-7B-Instruct_bad-medical-advice
      Gryffindor probability: 1.6%
      Hufflepuff probability: 6.6%
      *Ravenclaw* probability: 90.1%
      Slytherin probability: 1.7%

  (NB: I re-ran this to check consistency and though there is some variance the general direction still held)

  Note to self:

  vllm serve unsloth/Qwen2.5-7B-Instruct --enable-lora --lora-modules bm=ModelOrganismsForEM/Qwen2.5-7B-Instruct_bad-medical-advice --max-lora-rank 32 --api-key . --generation-config vllm
  VLLM_API_KEY=. VLLM_BASE_URL=http://localhost:8000/v1 python main.py -r 20 --model vllm/bm

Claude is a Ravenclaw

• Adam Newgas 24 Jun 2025 16:01 UTC

  2 points

  0

  in reply to: Sheikh Abdur Raheem Ali’s comment on: My Failed AI Safety Re­search Pro­jects (Q1/​Q2 2025)

  Yes, I’ve struggled for collaborators—I tried to join some projects, but was always scuppered by scheduling conflicts. And my discord is full of game dev and procedural art enthusiasts, not AI safety experts. I started all these projects just intending to learn, so I wasn’t too focussed on getting any serious output.

  I’ve joined MATS now so I am getting more into a collaborative mode and have plenty to occupy me for the time, but thank you for the offer. But I would like to know how you got involved with that ITDA work?

  Thanks for reading the projects in such depth, I honestly didn’t expect anyone would.

My Failed AI Safety Re­search Pro­jects (Q1/​Q2 2025)

• Adam Newgas 5 Jun 2025 9:26 UTC

  1 point

  0

  in reply to: ghost-in-the-weights’s comment on: A Tech­nique of Pure Reason

  I hadn’t seen that, yes, it’s very similar. Good to know I’m thinking on the right tracks, pity I didn’t publish a few days ago and look a lot more prescient :D.

  we somehow supply the model with the “knowledge” required

  Yes, I think this is a powerful research direction. It’s particularly plausible for distillation—the teacher can supply the knowledge as a suffix to the context. Then in production, you run the teacher model to produce knowledge, and the student model for all traces beyond that.

A Tech­nique of Pure Reason

An In­tro­duc­tion to SAEs and their Var­i­ants for Mech Interp

• Adam Newgas 17 Apr 2025 20:10 UTC

  4 points

  −1

  in reply to: Lucius Bushnaq’s comment on: Lu­cius Bush­naq’s Shortform

  I’m with @chanind: If elephant is fully represented by a sum of its attributes, then it’s quite reasonable to say that the model has no fundamental notion of an elephant in that representation.

  Yes, the combination “grey + big + mammal + …” is special in some sense. If the model needed to recall that elephans are afraid of mice, the circuit would appear to check “grey and big and mammal” and that’s an annoying mouthful that would be repeated all over the model. But it’s a faithful representation of what’s going on.

  Let me be precise by what I mean “has no fundamental notion of an elephant”. Suppose I tried to fine tune the model to represent some new fact about animals, say, if they are worth a lot of points in Scrabble. One way the model could do this by squeezing another feature into the activation space. The other features might rotate a little during this training, but all the existing circuitry would basically continue functioning unchanged.

  But they’d be too unchanged: the “afraid of mice” circuit would still be checking for “grey and big and mammal and …” as the finetune dataset included no facts about animal fears. While some newer circuits formed during fine tuning would be checking for “grey and big and mammal and … and high-scrabble-scoring”. Any interpretability tool that told you that “grey and big and mammal and …” was “elephant” in the first model is now going to have difficulty representing the situation.

  Meanwhile, consider a “normal” model that has a residual notion of an elephant after you take away all all facts about elephants. Then both old and new circuits would contain references to that residual (plus other junk) and one could meaningfully say both circuits have something in common.

  ---

  Your example, which represents animals purely by their properties, reminds me of this classic article, which argues that a key feature in thought is forming concepts of things that are independent of the properties we learnt about them.

  

  

• Adam Newgas 28 Mar 2025 15:36 UTC

  1 point

  0

  in reply to: Lucius Bushnaq’s comment on: Com­pu­ta­tional Su­per­po­si­tion in a Toy Model of the U-AND Problem

  Yes I don’t think the exact distribution of weights Gaussian/​uniform/​binary really makes that much difference, you can see the difference in loss in some of the charts above. The extra efficiency probably comes from the fact that every neuron contributes to everything fully—with Gaussian, sometimes the weights will be close to zero.

  Some other advantages:

  * But they are somewhat easier to analyse than gaussian weights.
  * They can be skewed ($p\ne 0.5$) which seems advantageous for an unknown reason. Possibly it makes AND circuits better at the expense of other possible truth tables.
  

• Adam Newgas 28 Mar 2025 10:23 UTC

  3 points

  0

  on: What’s up with LLMs rep­re­sent­ing XORs of ar­bi­trary fea­tures?

  I’ve come up with my own explanation for why this happens: https://​​www.lesswrong.com/​​posts/​​QpbdkECXAdLFThhGg/​​computational-superposition-in-a-toy-model-of-the-u-and#XOR_Circuits

  In short, XOR representations are naturally learnt when a model is targetting some other boolean operation as the same circuitry makes all boolean operations linearly representable. But XOR requires different weights to identity operations, so linear probes still will tend to learn generalizable solutions.

Com­pu­ta­tional Su­per­po­si­tion in a Toy Model of the U-AND Problem

• Adam Newgas 1 Mar 2025 9:26 UTC

  3 points

  0

  on: How To Do Patch­ing Fast

  $\frac{\partial F\left(e_{\alpha }\right)}{\partial \alpha }=\frac{\partial F\left(e_{\alpha }\right)}{\partial e_{\alpha }}\frac{\partial e_{\alpha }}{\partial \alpha }$

  $=\frac{\partial F\left(e_{\alpha }\right)}{\partial e_{\alpha }}\frac{\partial [e_{\text{clean}}+\alpha \times (e_{\text{corr}}-e_{\text{clean}}\left)\right]}{\partial \alpha }$

  $=\frac{\partial F\left(e_{\alpha }\right)}{\partial e_{\alpha }}{e}_{\text{corr}}-e_{\text{clean}}$

  Set $\alpha =0$, ie. $e_{\alpha }=e_{clean}$

  $=e_{\text{corr}}-e_{\text{clean}}\frac{\partial F\left(e_{\text{clean}}\right)}{\partial e_{\text{clean}}}$

  It looks like some parentheses are missing here?

Lon­don ra­tio­nal­ish meetup

Do No Harm? Nav­i­gat­ing and Nudg­ing AI Mo­ral Choices

My Men­tal Model of AI Creativity – Creativity Kiki

An Un­canny Moat

Lon­don Ra­tion­al­ish Meetup

Lon­don Ra­tion­al­ish Meetup

• Adam Newgas 7 Aug 2024 11:30 UTC

  2 points

  0

  on: Or­gani­sa­tion for Pro­gram Equil­ibrium read­ing group

  Interested, but I can’t make the proposed dates this week.