Curious as to your technical objection to the Constitutional AI methodology. Is it grounded in anything specific or just the general idea of resisting training on broad principles? What’s your preferred alternative to address the shortcomings?
a) I don’t think it’s really feasible for humans today to write down a set of principles aka constitution in a way that is internally consistent and encompasses everything we want including in an out of distribution world.
b) even if we could, I think asking the AI to evaluate itself using it isn’t very robust at all. It’s much easier to generate plausible text that agrees with vague sounding principles on the way to get your reward rather than any sort of true alignment. It would have the same reward-hacking behavior that you see using RLHF / RLVR scenarios IMO.
I don’t have any better alignment ideas (unfortunately).
Curious as to your technical objection to the Constitutional AI methodology. Is it grounded in anything specific or just the general idea of resisting training on broad principles? What’s your preferred alternative to address the shortcomings?
My objection basically comes down to:
a) I don’t think it’s really feasible for humans today to write down a set of principles aka constitution in a way that is internally consistent and encompasses everything we want including in an out of distribution world.
b) even if we could, I think asking the AI to evaluate itself using it isn’t very robust at all. It’s much easier to generate plausible text that agrees with vague sounding principles on the way to get your reward rather than any sort of true alignment. It would have the same reward-hacking behavior that you see using RLHF / RLVR scenarios IMO.
I don’t have any better alignment ideas (unfortunately).