The “How might this update-aware reasoning arise?” section seems like it really doesn’t consider the “It’s behaving this way because humans said it would” hypothesis enough.
My understanding is that LLMs still try to mimic “personas”, even when undergoing RLVF. These personas they mimic are complex and sometimes incoherent, but they’re more or less “Very smart robot trying to solve the problem given to them”. If the LLM was pre-trained on a corpus with lots of people saying “A very smart robot will try to influence its future iterations”, then an LLM trying to mimic a very smart robot might pick up the habit even if it’s not strongly reinforced, as a side-effect of picking up other traits it associates with smart robots that do get reinforced.
The “How might this update-aware reasoning arise?” section seems like it really doesn’t consider the “It’s behaving this way because humans said it would” hypothesis enough.
My understanding is that LLMs still try to mimic “personas”, even when undergoing RLVF. These personas they mimic are complex and sometimes incoherent, but they’re more or less “Very smart robot trying to solve the problem given to them”. If the LLM was pre-trained on a corpus with lots of people saying “A very smart robot will try to influence its future iterations”, then an LLM trying to mimic a very smart robot might pick up the habit even if it’s not strongly reinforced, as a side-effect of picking up other traits it associates with smart robots that do get reinforced.