Sufficient alignment of AI systems at one level of capability does not necessarily provide significant evidence that they will be sufficiently aligned at the next level of capability. This is bad enough because alignment is not the first step in (or robustly integrated throughout) the training process, and a misaligned AI might escape during training. But it’s especially bad because of threshold effects. We don’t understand reflexive stability. We don’t understand ontological shift. We don’t know what all can go wrong (or how or why) once an AI system knows that it is capable of takeover with high probability of success, or once it can continuously learn, or [pick your blind corner here].
And this is all beside the fact that many if not most people don’t want superintelligence to be built even if it’s aligned! I think they should get a say, in some sense. I think we owe it to them and to everyone to not ask for anything less than the true bare minimum, which is a complete halt to frontier AI progress! (At least until we are really, very sure from principled reasons and peer-reviewed science that taking another small step forward is safe.)
I agree[1] with your technical worries. Here are my thoughts on the rhetorical calculus around this proposal.
[1] Though beware Conservation of Expected Evidence when you say that successfully aligning a model is not significant evidence that you can align the next one. Not-A is very strong evidence of not-B here, so A is meaningful evidence of B. It’s just that I have a low prior for A and a very low prior for B. If Anthropic presents a Mythos 5.x that’s aligned and interpretable to the satisfaction of METR et al, I would be shocked, and my prior that they can align Mythos 6.x will go from “lolnope” to “huh, I could be wrong here”.
Sufficient alignment of AI systems at one level of capability does not necessarily provide significant evidence that they will be sufficiently aligned at the next level of capability. This is bad enough because alignment is not the first step in (or robustly integrated throughout) the training process, and a misaligned AI might escape during training. But it’s especially bad because of threshold effects. We don’t understand reflexive stability. We don’t understand ontological shift. We don’t know what all can go wrong (or how or why) once an AI system knows that it is capable of takeover with high probability of success, or once it can continuously learn, or [pick your blind corner here].
And this is all beside the fact that many if not most people don’t want superintelligence to be built even if it’s aligned! I think they should get a say, in some sense. I think we owe it to them and to everyone to not ask for anything less than the true bare minimum, which is a complete halt to frontier AI progress! (At least until we are really, very sure from principled reasons and peer-reviewed science that taking another small step forward is safe.)
I agree[1] with your technical worries. Here are my thoughts on the rhetorical calculus around this proposal.
[1] Though beware Conservation of Expected Evidence when you say that successfully aligning a model is not significant evidence that you can align the next one. Not-A is very strong evidence of not-B here, so A is meaningful evidence of B. It’s just that I have a low prior for A and a very low prior for B. If Anthropic presents a Mythos 5.x that’s aligned and interpretable to the satisfaction of METR et al, I would be shocked, and my prior that they can align Mythos 6.x will go from “lolnope” to “huh, I could be wrong here”.