I hear a lot of people say that when you do continual learning or online training, the training-deployment distinction no longer makes sense (e.g., Dwarkesh). I think this isn’t true.
I think that even once you have continual learning, it’s fairly likely that there will still be a stage before internal deployment in which you train your AIs on tasks that aren’t directly useful, whose outputs you’ll just throw away. It’s plausible that this training stage will be large and determine a substantial fraction of the AIs’ properties and will be a meaningful point in which to run evaluations (even if it also makes sense to do continuous/periodic auditing).
Of course, eventually AI development will be very unrecognizable and may be more continuous, but continual learning doesn’t delineate that point.
I feel making train/deployment/eval distinctions is a bit too coarse, and started being too coarse when we started doing RL. I don’t see a principled difference between the online learning regime and the periods of RL labs do internally. What matters is the degree to which models are
Aware of the training objective they are/are not subject to
Aware of the control measures they’re subject to
Aware of what affordances they have
Aware of what aspects of their behavior people are watching, and what actions people will take as a result of making certain observations
(behavior here including cot thoughts, and interp signals)
A true online learner, by which I imagine AIs making online updates to their weights within a single trajectory. Complicates things because:
More risk of alignment techniques stopping working
More risk of rapid capability gain
More risk of interp techniques stopping working, and really any monitoring tool that wasn’t designed to co-evolve with the changing model.
But these are all major problems that surface in current RL runs. They would just be a lot worse.
I hear a lot of people say that when you do continual learning or online training, the training-deployment distinction no longer makes sense (e.g., Dwarkesh). I think this isn’t true.
I think that even once you have continual learning, it’s fairly likely that there will still be a stage before internal deployment in which you train your AIs on tasks that aren’t directly useful, whose outputs you’ll just throw away. It’s plausible that this training stage will be large and determine a substantial fraction of the AIs’ properties and will be a meaningful point in which to run evaluations (even if it also makes sense to do continuous/periodic auditing).
Of course, eventually AI development will be very unrecognizable and may be more continuous, but continual learning doesn’t delineate that point.
I feel making train/deployment/eval distinctions is a bit too coarse, and started being too coarse when we started doing RL. I don’t see a principled difference between the online learning regime and the periods of RL labs do internally. What matters is the degree to which models are
Aware of the training objective they are/are not subject to
Aware of the control measures they’re subject to
Aware of what affordances they have
Aware of what aspects of their behavior people are watching, and what actions people will take as a result of making certain observations
(behavior here including cot thoughts, and interp signals)
A true online learner, by which I imagine AIs making online updates to their weights within a single trajectory. Complicates things because:
More risk of alignment techniques stopping working
More risk of rapid capability gain
More risk of interp techniques stopping working, and really any monitoring tool that wasn’t designed to co-evolve with the changing model.
But these are all major problems that surface in current RL runs. They would just be a lot worse.