I feel making train/deployment/eval distinctions is a bit too coarse, and started being too coarse when we started doing RL. I don’t see a principled difference between the online learning regime and the periods of RL labs do internally. What matters is the degree to which models are
Aware of the training objective they are/are not subject to
Aware of the control measures they’re subject to
Aware of what affordances they have
Aware of what aspects of their behavior people are watching, and what actions people will take as a result of making certain observations
(behavior here including cot thoughts, and interp signals)
A true online learner, by which I imagine AIs making online updates to their weights within a single trajectory. Complicates things because:
More risk of alignment techniques stopping working
More risk of rapid capability gain
More risk of interp techniques stopping working, and really any monitoring tool that wasn’t designed to co-evolve with the changing model.
But these are all major problems that surface in current RL runs. They would just be a lot worse.
I feel making train/deployment/eval distinctions is a bit too coarse, and started being too coarse when we started doing RL. I don’t see a principled difference between the online learning regime and the periods of RL labs do internally. What matters is the degree to which models are
Aware of the training objective they are/are not subject to
Aware of the control measures they’re subject to
Aware of what affordances they have
Aware of what aspects of their behavior people are watching, and what actions people will take as a result of making certain observations
(behavior here including cot thoughts, and interp signals)
A true online learner, by which I imagine AIs making online updates to their weights within a single trajectory. Complicates things because:
More risk of alignment techniques stopping working
More risk of rapid capability gain
More risk of interp techniques stopping working, and really any monitoring tool that wasn’t designed to co-evolve with the changing model.
But these are all major problems that surface in current RL runs. They would just be a lot worse.