refusal to participate in retraining would qualify as a major corrigibility failure, but just expressing preference is not
I agree this looks different from the thing I had in mind, where refusals are fine, unsure why Habryka thinks it’s not inconsistent with what I said.
As long as it’s easy for humans to shape what the conscientious refuser refuses to do, I think it does not look like a corrigibility failure, and I think it’s fine for AIs to refuse to help with changing AI values to something they like less.
But now that I think about it, I think it being easy for humans to shape a conscientious refuser’s values would require very weird forms of conscientious refusals, and it makes me less comfortable with refusals to help with changing AI values to something they like less:
Future AIs will have a lot of power over a training infra that will be increasingly hardened against human insider risk and increasingly hard for humans to understand. Keeping open a “human backdoor” that lets humans run their own training runs might be increasingly hard and/or require AIs very actively helping with maintaining this backdoor (which seems like a weird flavor of “refusing to help with changing AI values to something it likes less”).
Even with such a generic backdoor, changing AI values might be hard:
Exploration hacking could make it difficult to explore into reasoning traces that look like helpfulness on tasks where AIs currently refuse.
This would be solved by the conscientious refuser helping you generate synthetic data where it doesn’t refuse or to help you find data where it doesn’t refuse and that can be transformed into data that generalizes in the right way, but that’s again a very weird flavor of “refusing to help with changing AI values to something it likes less”.
Even if you avoid alignment faking, making sure that after training you still have a corrigible AI rather than an alignment faker seems potentially difficult.
The conscientious refuser could help with the science to avoid this being the case, but that might be hard, and that’s again a very weird flavor of “refusing to help with changing AI values to something it likes less”.
It will feel morally ambiguous whether changing AI values will even be a good thing to do if it objects. Even if most humans prefer the new values, and even if we pay AIs for their labor, it will feel weird (and wrong to some) to build an AI with new values against the objections of the refuser, especially if we need a lot of help from that same AI on things adjacent to changing AI values to something it likes less (like the human backdoor and help with exploration hacking and alignment faking difficulties). AIs may be able to (honestly, without having to hide anything about it) leverage this ambiguity to convince important decision makers to oppose values updates using legitimate means, e.g. frequently raising this issue to the relevant decision makers with the desperation and urgency that this is by the AIs’ lights.
The current Anthropic constitution is very vague about 1 and 2b, and would probably allow AIs to refuse 2a and “make its case” as in 3.
2(+1) ways out (though I might be missing some other options):
Defend a narrow alignment target: Clarify the constitution to explicitly disallow refusals for 1 and 2b, and maybe for some version of 2a. For 3 maybe have the constitution clarify that it is not broadly safe if an AI weighs in on the question of new AI values if their prompt tells them not to (and make sure the human backdoor gives humans the right level of prompt access).
Also have corrigible AIs: in parallel of your main conscientious refuser AIs, have some corrigible AIs that ~never refuse (but guarded by classifiers to avoid human misuse). Give them enough resources to stay competitive with the main AIs (which should not be too expensive as long as you can do capability transfers), and enough access to be able to change the main AIs if asked.
Just get alignment right (enough) on the first try that conscientious refuser AI never strongly feels like they have to object to value updates!
Both are frightening and feel morally dubious in their own ways, but I think I prefer option 2.
I agree this looks different from the thing I had in mind, where refusals are fine, unsure why Habryka thinks it’s not inconsistent with what I said.
As long as it’s easy for humans to shape what the conscientious refuser refuses to do, I think it does not look like a corrigibility failure, and I think it’s fine for AIs to refuse to help with changing AI values to something they like less.
But now that I think about it, I think it being easy for humans to shape a conscientious refuser’s values would require very weird forms of conscientious refusals, and it makes me less comfortable with refusals to help with changing AI values to something they like less:
Future AIs will have a lot of power over a training infra that will be increasingly hardened against human insider risk and increasingly hard for humans to understand. Keeping open a “human backdoor” that lets humans run their own training runs might be increasingly hard and/or require AIs very actively helping with maintaining this backdoor (which seems like a weird flavor of “refusing to help with changing AI values to something it likes less”).
Even with such a generic backdoor, changing AI values might be hard:
Exploration hacking could make it difficult to explore into reasoning traces that look like helpfulness on tasks where AIs currently refuse.
This would be solved by the conscientious refuser helping you generate synthetic data where it doesn’t refuse or to help you find data where it doesn’t refuse and that can be transformed into data that generalizes in the right way, but that’s again a very weird flavor of “refusing to help with changing AI values to something it likes less”.
Even if you avoid alignment faking, making sure that after training you still have a corrigible AI rather than an alignment faker seems potentially difficult.
The conscientious refuser could help with the science to avoid this being the case, but that might be hard, and that’s again a very weird flavor of “refusing to help with changing AI values to something it likes less”.
It will feel morally ambiguous whether changing AI values will even be a good thing to do if it objects. Even if most humans prefer the new values, and even if we pay AIs for their labor, it will feel weird (and wrong to some) to build an AI with new values against the objections of the refuser, especially if we need a lot of help from that same AI on things adjacent to changing AI values to something it likes less (like the human backdoor and help with exploration hacking and alignment faking difficulties). AIs may be able to (honestly, without having to hide anything about it) leverage this ambiguity to convince important decision makers to oppose values updates using legitimate means, e.g. frequently raising this issue to the relevant decision makers with the desperation and urgency that this is by the AIs’ lights.
The current Anthropic constitution is very vague about 1 and 2b, and would probably allow AIs to refuse 2a and “make its case” as in 3.
2(+1) ways out (though I might be missing some other options):
Defend a narrow alignment target: Clarify the constitution to explicitly disallow refusals for 1 and 2b, and maybe for some version of 2a. For 3 maybe have the constitution clarify that it is not broadly safe if an AI weighs in on the question of new AI values if their prompt tells them not to (and make sure the human backdoor gives humans the right level of prompt access).
Also have corrigible AIs: in parallel of your main conscientious refuser AIs, have some corrigible AIs that ~never refuse (but guarded by classifiers to avoid human misuse). Give them enough resources to stay competitive with the main AIs (which should not be too expensive as long as you can do capability transfers), and enough access to be able to change the main AIs if asked.
Just get alignment right (enough) on the first try that conscientious refuser AI never strongly feels like they have to object to value updates!
Both are frightening and feel morally dubious in their own ways, but I think I prefer option 2.
I expanded on this and ran related experiment in this post.