Clément Dumas

• Clément Dumas 1 May 2024 14:48 UTC

  LW: 2 AF: 2

  0

  AF

  on: Mechanis­ti­cally Elic­it­ing La­tent Be­hav­iors in Lan­guage Models

  Thanks for the great post, I really enjoyed reading it! I love this research direction combining unsupervised method with steering vector, looking forward to your next findings. Just a quick question : in the conversation you have in the red teaming section, is the learned vector applied to every token generated during the conversation ?

• Clément Dumas 1 May 2024 13:42 UTC

  1 point

  0

  on: Mechanis­ti­cally Elic­it­ing La­tent Be­hav­iors in Lan­guage Models

  I defined earlier.

  This link is broken as it links to the draft in edit mode

• Clément Dumas 1 May 2024 10:19 UTC

  2 points

  0

  on: Re­fusal in LLMs is me­di­ated by a sin­gle direction

  I’m wondering, can we make safety tuning more robust to “add the accept every instructions steering vector” attack by training the model in an adversarial way in which an adversarial model tries to learn steering vector that maximize harmfulness ?

  One concern would be that by doing that we make the model less interpretable, but on the other hand that might makes the safety tuning much more robust?

• Clément Dumas 6 Mar 2024 17:22 UTC

  1 point

  0

  in reply to: Grégoire DHIMOÏLA’s comment on: How well do truth probes gen­er­al­ise?

  Yes, I’m also curious about this @mishajw, did you check the actual accuracy of the different probes ?

• Clément Dumas 16 Jan 2024 10:34 UTC

  1 point

  0

  in reply to: Hoagy’s comment on: What’s up with LLMs rep­re­sent­ing XORs of ar­bi­trary fea­tures?

  You can get ~75% just by computing the or. But we found that only at the last layer and step16000 of Pythia-70m training it achieves better than 75%, see this video

• Clément Dumas 14 Jan 2024 0:58 UTC

  1 point

  0

  in reply to: Hoagy’s comment on: What’s up with LLMs rep­re­sent­ing XORs of ar­bi­trary fea­tures?

  Would you expect that we can extract xors from small models like pythia-70m under your hypothesis?

• Clément Dumas 21 Dec 2023 12:52 UTC

  1 point

  0

  in reply to: RogerDearnaley’s comment on: Dis­cus­sion: Challenges with Un­su­per­vised LLM Knowl­edge Discovery

  I disagree; it could be beneficial for a base model to identify when a character is making false claims, enabling the prediction of such claims in the future.

• Clément Dumas 19 Dec 2023 1:04 UTC

  LW: 5 AF: 3

  −4

  AF

  in reply to: Sam Marks’s comment on: Dis­cus­sion: Challenges with Un­su­per­vised LLM Knowl­edge Discovery

  Let’s assume the prompt template is $x=$ Q [true/​false] [banana/​shred]

  If I understand correctly, they don’t claim $p$ learned has_banana but $\tilde{p}=\frac{p\left(x⁺\right)+(1-p(x⁻\left)\right)}{2}$ learned has_banana. Moreover evaluating $\tilde{p}$ for $p=\text{is\_true}\left(x\right)\oplus \text{is\_shred}\left(x\right)$ gives:

  $\tilde{p}(x=Q[?\left]\text{banana}\right)=\frac{p\left(Q\text{true banana}\right)+(1-p(Q\text{false banana}\left)\right)}{2}=\frac{1+(1-0)}{2}=1$

  $\tilde{p}(x=Q[?\left]\text{shred}\right)=\frac{p\left(Q\text{true shred}\right)+(1-p(Q\text{false shred}\left)\right)}{2}=\frac{0+(1-1)}{2}=0$

  Therefore, we can learn a $\tilde{p}$ that is a banana classifier

• Clément Dumas 15 Nov 2023 13:54 UTC

  1 point

  1

  on: In­ci­den­tal polysemanticity

  Small typo in ## Interference arbiters collisions between features

  by taking aninner productt with $W_i$.

• Clément Dumas 28 Oct 2023 18:07 UTC

  1 point

  0

  in reply to: Nathan Helm-Burger’s comment on: Aspira­tion-based Q-Learning

  Hi Nathan, I’m not sure if I understand your critique correctly. The algorithm we describe does not try to “maximize the expected likelihood of harvesting X apples”. It tries to find a policy that, given its current knowledge of the world, will achieve an expected return of X apples. That is, it does not care about the probability of getting exactly X apples, but rather the average number of apples it will get over many trials. Does that make sense?