The continuity argument gives something like an “alignment prior” based on previous models’ alignment, distinct from other sources of evidence from alignment assessments such as training data monitoring and behavioral auditing.
For example, Anthropic’s August Alignment Risk Report said (p39-40):
“Claim 3.1.3: Covered models were trained in a sufficiently similar way to prior models that the propensities of prior models provide some evidence about the propensities of covered models.”
“Similarity here is not well defined, but we believe that it is intuitive to aggregate changes in scale, architecture, training processes, training data, final model capabilities, and observed propensities to come up with a rough overall estimate of similarity between two models.”
An observation about the continuity argument is that it might be significantly undermined by a shift to alternative architectures, which will probably co-occur with changes in training processes. It seems likely that an architectural shift will also co-occur with changes in scale, final model capabilities, and observed propensities, i.e., the new model is the result of larger training runs, is significantly more capable than the previous models, and either differs in actual observed propensities or the observability of its propensities (e.g. is much less CoT monitorable) compared to previous models.
If and when an architectural shift happens alongside all these other changes[1], it seems that our alignment prior can no longer depend on continuity with previous models. Our alignment prior might still be supported by other factors (e.g. the absolute capability levels at which you expect highly egregious forms of misalignment to arise, or the specific training processes needed for this), but I expect it would be substantially weakened.
Since I believe that current alignment assessments importantly depend on priors, I worry that the loss of the continuity argument would cause a significant amount of additional uncertainty about alignment. If so, this seems like an underrated safety cost of switching to alternative architectures.
It’s interesting that other people’s alignment priors for Astra don’t seem weaker than for previous models. This can be explained by them not believing Astra’s architecture (and/or training processes, training scale etc.) is significantly different, not reflecting sufficiently on the continuity argument, or the continuity argument not being load-bearing for their beliefs. I’m curious if this seems right to people and what the right explanation is.
We don’t understand AI well enough to make continuity arguments about anything even now. Based on how fragile alignment midtraining is, it’s plausible to me that random changes in data or architecture will make models much more misaligned.
Isn’t the continuity argument implying that you don’t have to understand the AI well to be somewhat confident in it’s alignment properties? I think it’s plausible random changes in data make models more misaligned, but to the extent those changes look like previous changes, I wouldn’t predict the alignment properties get significantly worse.
Architecture changes seem like a much bigger deal and I’d be less likely to trust continuity under these.
You can use some Laplace rule like “because 5 out of 5 models have been aligned so far, the next one has a 6⁄7 chance of being aligned”, but this is too weak a bound to put in safety cases. For one thing, 1⁄7 is too high a probability and we need 99 safe models to drive it below 1%. For another, if you expect misalignment to happen above some capability level, you can’t just keep using this argument when scaling up.
This doesn’t make continuity arguments useless but we do need to rely on a better understanding.
It’s interesting that other people’s alignment priors for Astra don’t seem weaker than for previous models. This can be explained by them not believing Astra’s architecture (and/or training processes, training scale etc.) is significantly different, not reflecting sufficiently on the continuity argument, or the continuity argument not being load-bearing for their beliefs. I’m curious if this seems right to people and what the right explanation is.
In my mind it’s mostly the training procedures not being significantly different. If the training procedures used to train models with scaled recurrent depth looked significantly dissimilar to previous training procedures, and/or the model was much bigger than previous models, I’d be much more uncertain about it’s alignment properties.
I’m a bit worried about frog boiling because this prior is so strong, but it seems good that we are exploring training procedure changes faster than architecture changes. For instance, I think we got lucky that OpenAI scaled up multi-agent RL (a training procedure change) before making significant architecture changes that could obfuscate the involved reasoning traces. Maybe it is easier to make less scary changes than it is to make more scary changes, or if it’s not easier, there is something else holding the labs back from making those changes (taboo’s, safety researchers, etc.)
Another possible view on this (with implications I don’t overall endorse) is that the architectural overhang seems scarier if one had hoped to rely on the continuity argument. That is, if we had explored architectural changes faster in the past, then we might have had more time to build up a continuity case using previous models of this architecture being aligned (assuming there were good ways to measure the weaker models’ propensities well), and it might be less scary to have a scaled-up model of this architecture now.
(There are probably stronger effects in the same direction, e.g., we would have had more time to iterate on safety techniques specific to the new architecture.)
I continue to insist that we have a threshold problem, not just a speed problem. We currently have no way to constrain what will happen when AI becomes strictly superhuman at creating better AI, or when it gains the option to permanently escape human oversight or disempower humanity. We have no guarantees of the degree of generalization from our intent at those thresholds. They are opaque walls to us.
The continuity argument fails at all of these steps, not just new architectures. But if AI does an increasing amount of the AI research, we’ll also have no way of preventing the emergence of new architectures!
The emergent behavior of agent swarms was a relatively transparent threshold, and even that showed that we are just blundering forward with no idea what we are doing.
The continuity argument gives something like an “alignment prior” based on previous models’ alignment, distinct from other sources of evidence from alignment assessments such as training data monitoring and behavioral auditing.
For example, Anthropic’s August Alignment Risk Report said (p39-40):
“Claim 3.1.3: Covered models were trained in a sufficiently similar way to prior models that the propensities of prior models provide some evidence about the propensities of covered models.”
“Similarity here is not well defined, but we believe that it is intuitive to aggregate changes in scale, architecture, training processes, training data, final model capabilities, and observed propensities to come up with a rough overall estimate of similarity between two models.”
An observation about the continuity argument is that it might be significantly undermined by a shift to alternative architectures, which will probably co-occur with changes in training processes. It seems likely that an architectural shift will also co-occur with changes in scale, final model capabilities, and observed propensities, i.e., the new model is the result of larger training runs, is significantly more capable than the previous models, and either differs in actual observed propensities or the observability of its propensities (e.g. is much less CoT monitorable) compared to previous models.
If and when an architectural shift happens alongside all these other changes[1], it seems that our alignment prior can no longer depend on continuity with previous models. Our alignment prior might still be supported by other factors (e.g. the absolute capability levels at which you expect highly egregious forms of misalignment to arise, or the specific training processes needed for this), but I expect it would be substantially weakened.
Since I believe that current alignment assessments importantly depend on priors, I worry that the loss of the continuity argument would cause a significant amount of additional uncertainty about alignment. If so, this seems like an underrated safety cost of switching to alternative architectures.
It’s interesting that other people’s alignment priors for Astra don’t seem weaker than for previous models. This can be explained by them not believing Astra’s architecture (and/or training processes, training scale etc.) is significantly different, not reflecting sufficiently on the continuity argument, or the continuity argument not being load-bearing for their beliefs. I’m curious if this seems right to people and what the right explanation is.
We don’t understand AI well enough to make continuity arguments about anything even now. Based on how fragile alignment midtraining is, it’s plausible to me that random changes in data or architecture will make models much more misaligned.
Isn’t the continuity argument implying that you don’t have to understand the AI well to be somewhat confident in it’s alignment properties? I think it’s plausible random changes in data make models more misaligned, but to the extent those changes look like previous changes, I wouldn’t predict the alignment properties get significantly worse.
Architecture changes seem like a much bigger deal and I’d be less likely to trust continuity under these.
You can use some Laplace rule like “because 5 out of 5 models have been aligned so far, the next one has a 6⁄7 chance of being aligned”, but this is too weak a bound to put in safety cases. For one thing, 1⁄7 is too high a probability and we need 99 safe models to drive it below 1%. For another, if you expect misalignment to happen above some capability level, you can’t just keep using this argument when scaling up.
This doesn’t make continuity arguments useless but we do need to rely on a better understanding.
In my mind it’s mostly the training procedures not being significantly different. If the training procedures used to train models with scaled recurrent depth looked significantly dissimilar to previous training procedures, and/or the model was much bigger than previous models, I’d be much more uncertain about it’s alignment properties.
I’m a bit worried about frog boiling because this prior is so strong, but it seems good that we are exploring training procedure changes faster than architecture changes. For instance, I think we got lucky that OpenAI scaled up multi-agent RL (a training procedure change) before making significant architecture changes that could obfuscate the involved reasoning traces. Maybe it is easier to make less scary changes than it is to make more scary changes, or if it’s not easier, there is something else holding the labs back from making those changes (taboo’s, safety researchers, etc.)
Another possible view on this (with implications I don’t overall endorse) is that the architectural overhang seems scarier if one had hoped to rely on the continuity argument. That is, if we had explored architectural changes faster in the past, then we might have had more time to build up a continuity case using previous models of this architecture being aligned (assuming there were good ways to measure the weaker models’ propensities well), and it might be less scary to have a scaled-up model of this architecture now.
(There are probably stronger effects in the same direction, e.g., we would have had more time to iterate on safety techniques specific to the new architecture.)
I continue to insist that we have a threshold problem, not just a speed problem. We currently have no way to constrain what will happen when AI becomes strictly superhuman at creating better AI, or when it gains the option to permanently escape human oversight or disempower humanity. We have no guarantees of the degree of generalization from our intent at those thresholds. They are opaque walls to us.
The continuity argument fails at all of these steps, not just new architectures. But if AI does an increasing amount of the AI research, we’ll also have no way of preventing the emergence of new architectures!
The emergent behavior of agent swarms was a relatively transparent threshold, and even that showed that we are just blundering forward with no idea what we are doing.