if the selection of memories happens in clear rounds where memories are evaluated according to rewards, then I don’t see clear structural differences between training and deployment.
Isn’t another structural difference that in deployment, this round structure may be happening many times in parallel, leading to increased variance in deployed models and increased chance that one of the models is misaligned?
E.g. say we have some sort of continual learning where deployed models specialize over the timeline of weeks-months to different task distributions. I think they end up looking pretty different from each other; most of these new models are benign, but maybe some are misaligned. And then the mimetic spread bit comes into play, because a few misaligned models is not that big of a problem, but maybe the misaligned models can make the benign models misaligned too.
Isn’t another structural difference that in deployment, this round structure may be happening many times in parallel, leading to increased variance in deployed models and increased chance that one of the models is misaligned?
E.g. say we have some sort of continual learning where deployed models specialize over the timeline of weeks-months to different task distributions. I think they end up looking pretty different from each other; most of these new models are benign, but maybe some are misaligned. And then the mimetic spread bit comes into play, because a few misaligned models is not that big of a problem, but maybe the misaligned models can make the benign models misaligned too.