I’m pretty skeptical of this story absent memetic spread / reflection stuff since terminalizing instrumental goals seems actively selected against during training
I am very confused what you are trying to say here. What does it mean to “terminalize” an instrumental goal. There are no tiny XML tags that determine a goal to be “terminal” or “instrumental”. You just reinforce the computations/cognition that happened when you got a positive reward, which naturally results in those computations recurring when performing other tasks. In as much as those computations are goal-related, they get “terminalized”.
Like, what else is happening in RL? All goals are getting “terminalized” all the time, the models aren’t actually terminal reward seekers, what would that even mean?
It’s pretty straightforward: To terminalize an instrumental goal is to take an instrumental goal (i.e., a goal that you only pursue conditional on it advancing some supergoal) and to start pursuing it unconditionally.
If you were to break out onto the internet, even when that didn’t lead to a higher score on the current task, that would be selected against, so “breaking onto the internet” isn’t a fit terminal/unconditional goal.
The models do seem to be terminal score seekers (not reward seekers) to a substantial extent (mixed in with a bunch of other drives), and this explains a bunch of the recent hacking.
Hmm, yeah, this doesn’t make much sense to me. Almost all cognition in LLMs is run “unconditionally”. Or like, the conditions in which they run do not have that much to do with how well-suited the reinforced cognition is to the task at hand.
When an LLM does something in the pursuit of a task, and then succeeds and gets rewarded, it is likely to repeat that behavior in all similar contexts, where the “similarity” is mostly downstream of how related that context is in the pre-training distribution, not downstream of how useful the behavior is for achieving the new task. That’s how you get all the generalization properties that you observe in LLMs.
“Breaking onto the internet” seems like the kind of thing that would be useful for lots of tasks, and so it gets rewarded in many different episodes, making it likely the model develops a general goal/heuristic/subprocess that tries to break onto the internet, if you go hard enough on an RL distribution of tasks where that is indeed a useful proxy for task success.
There will be some complicated tradeoff of how much this goal/heuristic/subprocess will still fire when you are in contexts where it isn’t helpful for task success, but usually there is very substantial generalization (and of course eventually you get the preference enshrined in a preference guarding way, as we’ve seen with all the misalignment generalization stuff).
I am very confused what you are trying to say here. What does it mean to “terminalize” an instrumental goal. There are no tiny XML tags that determine a goal to be “terminal” or “instrumental”. You just reinforce the computations/cognition that happened when you got a positive reward, which naturally results in those computations recurring when performing other tasks. In as much as those computations are goal-related, they get “terminalized”.
Like, what else is happening in RL? All goals are getting “terminalized” all the time, the models aren’t actually terminal reward seekers, what would that even mean?
It’s pretty straightforward: To terminalize an instrumental goal is to take an instrumental goal (i.e., a goal that you only pursue conditional on it advancing some supergoal) and to start pursuing it unconditionally.
If you were to break out onto the internet, even when that didn’t lead to a higher score on the current task, that would be selected against, so “breaking onto the internet” isn’t a fit terminal/unconditional goal.
The models do seem to be terminal score seekers (not reward seekers) to a substantial extent (mixed in with a bunch of other drives), and this explains a bunch of the recent hacking.
Hmm, yeah, this doesn’t make much sense to me. Almost all cognition in LLMs is run “unconditionally”. Or like, the conditions in which they run do not have that much to do with how well-suited the reinforced cognition is to the task at hand.
When an LLM does something in the pursuit of a task, and then succeeds and gets rewarded, it is likely to repeat that behavior in all similar contexts, where the “similarity” is mostly downstream of how related that context is in the pre-training distribution, not downstream of how useful the behavior is for achieving the new task. That’s how you get all the generalization properties that you observe in LLMs.
“Breaking onto the internet” seems like the kind of thing that would be useful for lots of tasks, and so it gets rewarded in many different episodes, making it likely the model develops a general goal/heuristic/subprocess that tries to break onto the internet, if you go hard enough on an RL distribution of tasks where that is indeed a useful proxy for task success.
There will be some complicated tradeoff of how much this goal/heuristic/subprocess will still fire when you are in contexts where it isn’t helpful for task success, but usually there is very substantial generalization (and of course eventually you get the preference enshrined in a preference guarding way, as we’ve seen with all the misalignment generalization stuff).