Anthropic Fellow & NYU PhD student
andyqhan
Karma: 43
How’s it going? Reinforcement learning in language models recruits a functional welfare axis
For what it’s worth, I think that Justis hits the nail on the head with “I think probably under current conditions, broken English is less of a red flag for people than LLM-ese.” In such a global language as English, people naturally give slack. (Also, non-native speakers are kind of in an adversarial situation with LLM-ese, since it’s harder to detect when you aren’t as immersed in standard American/British English.)
Concrete example: my parents, whose English is fairly weak, always say that one of the nice things about America is that people are linguistically generous. They illustrate it like this: “In our country, if people can’t understand you, they think it’s your fault. In America, they think it’s theirs.” I think the same is true of the internet, especially somewhere like LessWrong.
On a practical note, I think spellcheckers like those in Docs and Word are sufficient for these contexts. In academic writing or whatever, when standard English serves more of a signaling function, it’s trickier.
Hi Edward, sorry I missed this! Thanks for the comment and the questions.
We primarily include the Path-final trajectories in order to test for antiparallelism, as you say. The Path-final activations lie somewhere between the Mold- and Gold-final trajectories’ activations, although it’s more of a triangle than a straight line. We run the experiment you suggest in (Appendix C)[https://functionalwelfare.com/appendix/c/#further-geometric-analyses] (the latter half of it) — I think C.7 answers your question (the vectors are antiparallel in raw activations as well), but I think C.8 is pretty neat too (instead of v_Gold = a_Gold—mean(a_Mold, a_Path), we do v_Gold = a_Gold—a_Path, and find that antiparallelism holds).
I expected this to be the case, and designed some of the sentiment questions around it. The results are unclear and I opted not to include an analysis in the paper because I didn’t quite know how to interpret it. I think that it’s a null result: I don’t think that the model “likes” the Gold emoji out of distribution. However, we didn’t conduct any rigorous study of this, and your suggestion of seeing whether the assistant would predict it would seek Gold is very interesting.
Ah yes. The short answer is that I don’t know, but I do think that the experiment you suggest should separate high from low reward trajectories. I’m going to run that now actually, shouldn’t be too hard. I’m also super interested in how the FW axis tracks stuff out of distribution — we don’t have very broad OOD results in the paper.
Feel free to reach out if you want to chat!