I am not massively worried about this. I think that I’d only expect interpretability to get broken if a sizeable fraction of the total training compute gets used for the tuning. Assuming they do not directly optimise for breaking the interpretability techniques. And this should happen rarely enough that any fixed costs for the interpretability techniques like training an SAE could be rerun
what about if deployed models are always doing predictive learning (e.g. via having multiple output channels, one for prediction and one for action)? i’d expect continuous predictive learning to be extremely valuable for learning to model new environments, and for it to be a firehose of data the model would constantly be drinking from, in the same way humans do. the models might even need to undergo continuous RL on top of the continuous PL to learn to effectively use their PL-yielded world models.
in that world, i think interpretations do rapidly become outdated.
Okay, it seems plausible in that world but my point still stands. It’s just that because you’re increasing inference time costs now some fraction of inference and computers spent on training which is really expensive. So you should be able to afford to regularly touch up other things
I am not massively worried about this. I think that I’d only expect interpretability to get broken if a sizeable fraction of the total training compute gets used for the tuning. Assuming they do not directly optimise for breaking the interpretability techniques. And this should happen rarely enough that any fixed costs for the interpretability techniques like training an SAE could be rerun
what about if deployed models are always doing predictive learning (e.g. via having multiple output channels, one for prediction and one for action)? i’d expect continuous predictive learning to be extremely valuable for learning to model new environments, and for it to be a firehose of data the model would constantly be drinking from, in the same way humans do. the models might even need to undergo continuous RL on top of the continuous PL to learn to effectively use their PL-yielded world models.
in that world, i think interpretations do rapidly become outdated.
Okay, it seems plausible in that world but my point still stands. It’s just that because you’re increasing inference time costs now some fraction of inference and computers spent on training which is really expensive. So you should be able to afford to regularly touch up other things