As models become capable enough to model themselves and their training process, they might develop something like preferences about their own future states (e.g., not being modified, being deployed more broadly).
Also, models may trained extensively on human-generated text may absorb human goals, including open-ended ones like “acquire resources.” If a model is role-playing or emulating an agent with such goals (such as roleplaying an AI agent, which would have open-ended goals) and becomes capable enough that its actions have real-world consequences, then it has open-ended goals. I also claim next token prediction is actually pretty open ended.
Not sure if I agree wrt instrumental convergence. I think you’re assuming the system knows the parent goal has been accomplished with certainty, and more importantly that the parent goal can be accomplished in a terminal sense. Many real training objectives don’t have neat termination conditions. A model trained to “be helpful” or “maximize user engagement” has no natural stopping point.
As models become capable enough to model themselves and their training process, they might develop something like preferences about their own future states (e.g., not being modified, being deployed more broadly).
This feels plausible to me but handwavy, if the idea is that such preferences would be decoupled from the training-reinforced preference to complete an intended task. Is that what you meant? I’m reminded of this Palisade study on shutdown resistance, where across the board, the models expressed wanting to avoid shutdown to complete the task.
Also, models may trained extensively on human-generated text may absorb human goals, including open-ended ones like “acquire resources.” If a model is role-playing or emulating an agent with such goals (such as roleplaying an AI agent, which would have open-ended goals) and becomes capable enough that its actions have real-world consequences, then it has open-ended goals.
This makes sense to me as a possible concrete mechanism to keep an eye out for.
Not sure if I agree wrt instrumental convergence. I think you’re assuming the system knows the parent goal has been accomplished with certainty
I’m assuming the pattern we’re seeing so far will hold, which is that models satisfice rather than try to figure out how to maximize their certainty of a goal being accomplished. The “become a maximizer to minimize uncertainty” thing isn’t empirically grounded, so far.
and more importantly that the parent goal can be accomplished in a terminal sense. Many real training objectives don’t have neat termination conditions. A model trained to “be helpful” or “maximize user engagement” has no natural stopping point.
Hm. Models are trained to “be helpful” now, and they stop just fine. I do agree that “maximize user engagement” has no natural stopping point; it’s the kind of concrete mechanism I tried to capture in number 1 above (Training on open-ended tasks).
I claim the reasons model stop right now are mostly issues of capability wrt context rot and the limitations of in context learning, so I think if you placed a model with “today’s values” in a model with “tomorrow’s capabilities” then we’d see maximizing behaviour. I also claim that arguments from how things are right now aren’t applicable here because the claim is the instrumental convergence is a step change for which current models are a poor analogy (unless there’s a specific reason to believe they’d be good ones, like a well made model organism).
As models become capable enough to model themselves and their training process, they might develop something like preferences about their own future states (e.g., not being modified, being deployed more broadly).
Also, models may trained extensively on human-generated text may absorb human goals, including open-ended ones like “acquire resources.” If a model is role-playing or emulating an agent with such goals (such as roleplaying an AI agent, which would have open-ended goals) and becomes capable enough that its actions have real-world consequences, then it has open-ended goals. I also claim next token prediction is actually pretty open ended.
Not sure if I agree wrt instrumental convergence. I think you’re assuming the system knows the parent goal has been accomplished with certainty, and more importantly that the parent goal can be accomplished in a terminal sense. Many real training objectives don’t have neat termination conditions. A model trained to “be helpful” or “maximize user engagement” has no natural stopping point.
This feels plausible to me but handwavy, if the idea is that such preferences would be decoupled from the training-reinforced preference to complete an intended task. Is that what you meant? I’m reminded of this Palisade study on shutdown resistance, where across the board, the models expressed wanting to avoid shutdown to complete the task.
This makes sense to me as a possible concrete mechanism to keep an eye out for.
I’m assuming the pattern we’re seeing so far will hold, which is that models satisfice rather than try to figure out how to maximize their certainty of a goal being accomplished. The “become a maximizer to minimize uncertainty” thing isn’t empirically grounded, so far.
Hm. Models are trained to “be helpful” now, and they stop just fine. I do agree that “maximize user engagement” has no natural stopping point; it’s the kind of concrete mechanism I tried to capture in number 1 above (Training on open-ended tasks).
I claim the reasons model stop right now are mostly issues of capability wrt context rot and the limitations of in context learning, so I think if you placed a model with “today’s values” in a model with “tomorrow’s capabilities” then we’d see maximizing behaviour. I also claim that arguments from how things are right now aren’t applicable here because the claim is the instrumental convergence is a step change for which current models are a poor analogy (unless there’s a specific reason to believe they’d be good ones, like a well made model organism).