Arthur Conmy

• Arthur Conmy 20 Mar 2023 22:48 UTC

  3 points

  0

  in reply to: gwern’s comment on: RLHF does not ap­pear to differ­en­tially cause mode-collapse

  That’s true, I think the pretraining gradients training choice probably has more effect on the end model than the overfitting SFT model they start PPO with.

  Huh, but Mysteries of mode collapse (and the update) were published before td-003 was released? How would you have ended up reading a post claiming td-002 was RLHF-trained when td-003 existed?
  
  Meta note: it’s plausibly net positive that all the training details of these models has been obfuscated, but it’s frustrating how much energy has been sunk into speculation on The Way Things Work Inside OpenAI.

• Arthur Conmy 20 Mar 2023 21:39 UTC

  1 point

  0

  in reply to: gwern’s comment on: RLHF does not ap­pear to differ­en­tially cause mode-collapse

  I wasn’t trying to say mode collapse results were wrong! I collected these results before finding crisper examples of mode collapse that I could build a useful interpretability project on. I also agree with the remarks made about the difficulty of measuring this phenomena. I indeed tried to use the OpenAI embeddings model to encode the various completions and then hopefully have the Euclidean distance be informative, but it seemed to predict large distances for similar completions so I gave up. I also made a consistent color scheme and compared code-davinci, thanks for those suggestions.

  I don’t get the impression that RLHF needs hacks to prevent mode collapse: the InstructGPT reports overfitting leading to better human-rater feedback, and the Anthropic HH paper mentions in passing that the KL penalty may be wholly irrelevant (!).

  I’m not sure how to interpret the evidence from your first paragraph. You suggest that td-003 mode collapses where td-002 is perfectly capable. So you believe that both td-002 and td-003 mode collapse, in disjoint cases (given the examples from the original mode collapse post)?

• Arthur Conmy 20 Mar 2023 17:59 UTC

  4 points

  0

  in reply to: Erik Jenner’s comment on: RLHF does not ap­pear to differ­en­tially cause mode-collapse

  Thanks—this doesn’t seem to change observations much, except there doesn’t seem to be a case where this model has starkly the lowest entropy, as we found with davinci

  EDIT: I added code-davinci-002 as the main focus of the post, thanks!

RLHF does not ap­pear to differ­en­tially cause mode-collapse

• Arthur Conmy 19 Mar 2023 16:48 UTC

  1 point

  0

  on: The Sin­gu­lar Value De­com­po­si­tions of Trans­former Weight Ma­tri­ces are Highly Interpretable

  its top singular vector encodes what we think are the *least* frequent tokens.

  I spot “GoldMagikarp” and “Skydragon”—we now know these are indeed very infrequent tokens! This was a good evidence for SolidGoldMagikarp lurking in plain sight : )

• Arthur Conmy 19 Mar 2023 10:44 UTC

  LW: 1 AF: 1

  0

  AF

  in reply to: Raemon’s comment on: “Pub­lish or Per­ish” (a quick note on why you should try to make your work leg­ible to ex­ist­ing aca­demic com­mu­ni­ties)

  I think this point was really overstated. I get the impression the rejected papers were basically turned into the arXiv format as fast as possible and so it was easy for the mods to tell this. However, I’ve seen submissions to cs.LG like this and this that are clearly from the alignment community. These posts are also not stellar by standards of preprint formatting, and were not rejected, apparently

• Arthur Conmy 6 Mar 2023 14:35 UTC

  3 points

  1

  in reply to: Quintin Pope’s comment on: The Waluigi Effect (mega-post)

  I had the identical reaction that the statement of this effect was a bizarre framing. @afspies’s comment was helpful—I don’t think the claim is as bizarre now.
  
  (though overall I don’t think this post is a useful contribution because it is more likely to confuse than to shed light on LMs)

• Arthur Conmy 5 Mar 2023 23:50 UTC

  4 points

  3

  in reply to: afspies’s comment on: The Waluigi Effect (mega-post)

  I meant your first point.
  
  Regarding the claim that finetuning on data with property $P$ will lead models to ‘understand’ (scare-quotes omitted from now on...) both $P$ and not $P$ better, thanks. I see better where the post is coming from.
  
  However, I don’t necessarily think that we get the easier elicitation of not $P$. There are reasons to believe finetuning is simply resteering the base model and not changing its understanding at all. For example, there are far more training steps in pretraining vs. finetuning. Even if finetuning is shaping a model’s understanding of $P$, in an RLHF setup you’re generally seeing two responses, one with less $P$ and one with more $P$, and I’m not sure that I buy that the model’s inclination to output not $P$ responses can increase given there are no gradients from not $P$ cases. There are in red-teaming setups though and I think the author should register predictions in advance and then blind test various base models and finetuned models for the Waluigi Effect.

• Arthur Conmy 5 Mar 2023 13:52 UTC

  4 points

  0

  in reply to: Lauro Langosco’s comment on: Causal Scrub­bing: a method for rigor­ously test­ing in­ter­pretabil­ity hy­pothe­ses [Red­wood Re­search]

  It is open sourced here and there is material from REMIX to get used to the codebase here

• Arthur Conmy 5 Mar 2023 10:44 UTC

  39 points

  18

  on: The Waluigi Effect (mega-post)

  The Waluigi Effect: After you train an LLM to satisfy a desirable property $P$, then it’s easier to elicit the chatbot into satisfying the exact opposite of property $P$.

  
  I’ve tried several times to engage with this claim, but it remains dubious to me and I didn’t find the croissant example enlightening.
  
  Firstly, I think there is weak evidence that training on properties makes opposite behavior easier to elicit. I believe this claim is largely based on the bing chat story, which may have these properties due to bad finetuning rather than because these finetuning methods cause the Waluigi effect. I think ChatGPT is an example of finetuning making these models more robust to prompt attacks (example).
  
  Secondly (and relatedly) I don’t think this article does enough to disentangle the effect of capability gains from the Waluigi effect. As models become more capable both in pretraining (understanding subtleties in language better) and in finetuning (lowering the barrier of entry for the prompting required to get useful outputs), they will get better at being jailbroken by stranger prompts.

OpenAI in­tro­duce ChatGPT API at 1/​10th the pre­vi­ous $/​token

• Arthur Conmy 18 Feb 2023 23:19 UTC

  LW: 1 AF: 1

  0

  AF

  on: EIS VII: A Challenge for Mechanists

  Does the “ground truth” shows the correct label function on 100% of the training and test data? If so, what’s the relevance of the transformer which imperfectly implements the label function?

• Arthur Conmy 6 Feb 2023 5:28 UTC

  12 points

  0

  in reply to: Nate Showell’s comment on: SolidGoldMag­ikarp (plus, prompt gen­er­a­tion)

  This is because of tokenization. Tutorial about BPE (which OpenAI use) is here. Specifically in this case:

  

• Arthur Conmy 31 Jan 2023 18:03 UTC

  3 points

  0

  on: Peter Thiel on Tech­nolog­i­cal Stag­na­tion and Out of Touch Rationalists

  It seems like a cached speech from him. He echoes the same words at the Oxford Union earlier this month. I’m unsure how much this needs updating on. He constantly pauses and is occasionally inflammatory so my impression was he was measuring his words carefully for the audience.

• Arthur Conmy 9 Jan 2023 7:33 UTC

  2 points

  0

  in reply to: rpglover64’s comment on: A Mechanis­tic In­ter­pretabil­ity Anal­y­sis of Grokking

  You can observe “double descent” of test loss curves in the grokking setting, and there is “grokking” of test set performance as model dimension is increased, as this paper points out

• Arthur Conmy 7 Jan 2023 2:05 UTC

  1 point

  0

  on: We may be able to see sharp left turns coming

  This post implies that we may be able to extrapolate the log-prob likelihood of deception, situational awareness, or power-seeking behavior in future models.
  
  Do you expect that models’ stated desires for these behaviors are the metrics that we should measure? Or that when models become more powerful, the metric to use will be obvious? I generally agree with the observation that capabilities tend to improve more smoothly than early alignment work predicted, but I am skeptical that we will know what is relevant to measure.

• Arthur Conmy 2 Jan 2023 22:49 UTC

  1 point

  0

  on: Pre­dict­ing GPU performance

  The criticism here implies that one of the most important factors in modelling the end of Moore’s law is whether we’re running out of ideas (which the poster thinks is true). Do you think your models capture the availability of new ideas?

• Arthur Conmy 21 Dec 2022 16:14 UTC

  16 points

  0

  in reply to: Eli Tyre’s comment on: Sazen

  On one hand wikipedia suggests Jewish astronomers saw the three tail stars as cubs. But at the same time, it suggests several ancient civilizations independently saw Ursa Major as a bear. Also confused.

• Arthur Conmy 20 Dec 2022 19:24 UTC

  2 points

  0

  on: Will re­search in AI risk jinx it? Con­se­quences of train­ing AI on AI risk arguments

  I would recommend reading the top comment on https://​​forum.effectivealtruism.org/​​posts/​​WPEwFS5KR3LSqXd5Z/​​why-do-we-post-our-ai-safety-plans-on-the-internet

• Arthur Conmy 6 Dec 2022 5:57 UTC

  0 points

  0

  on: Who are some promi­nent rea­son­able peo­ple who are con­fi­dent that AI won’t kill ev­ery­one?

  Yann LeCun contra instrumental convergence
  
  EDIT: oops!