This seems like a crucial topic, since relying on persona training for safety seems central to the default plan for alignment. So I appreciate all analysis in this direction.
This is not intended to sound optimistic. I am neither optimistic nor pessimistic, and I think others should broaden their uncertainties on this topic on average. I think it’s quite complex and the analyses we’ve done so far are not remotely adequate.
I don’t think this article meets the optimistic case at its strong points.
The argument here seems to be of the form: This might go wrong. Therefore it will go wrong. I think that’s true, but also it might go right. Betting the future on such vague logic would be tragic, so refining these arguments seems pretty critical.
It’s not clear to me personas developed with existing datasets wouldn’t generalize to smarter versions of the system. There doesn’t need to be a crisp natural abstraction of “the good” for this to work. Claude has abstract representations of the stuff it cares about. Those might generalize adequately to survive any ontological shifts. Or they might not.
If it were really like taking two shots in the dark (human values and then ASI values), then the counting argument works and there’s little chance of alignment. But the effort is very much intended to be guided, to not be done in the dark. We are trying very hard to aim the model’s alignment at human values.
This seems like a crucial topic, since relying on persona training for safety seems central to the default plan for alignment. So I appreciate all analysis in this direction.
This is not intended to sound optimistic. I am neither optimistic nor pessimistic, and I think others should broaden their uncertainties on this topic on average. I think it’s quite complex and the analyses we’ve done so far are not remotely adequate.
I don’t think this article meets the optimistic case at its strong points.
The argument here seems to be of the form: This might go wrong. Therefore it will go wrong. I think that’s true, but also it might go right. Betting the future on such vague logic would be tragic, so refining these arguments seems pretty critical.
It’s not clear to me personas developed with existing datasets wouldn’t generalize to smarter versions of the system. There doesn’t need to be a crisp natural abstraction of “the good” for this to work. Claude has abstract representations of the stuff it cares about. Those might generalize adequately to survive any ontological shifts. Or they might not.
If it were really like taking two shots in the dark (human values and then ASI values), then the counting argument works and there’s little chance of alignment. But the effort is very much intended to be guided, to not be done in the dark. We are trying very hard to aim the model’s alignment at human values.
I analyzed many ways this could go wrong in LLM AGI may reason about its goals and discover misalignments by default. But doing that careful analysis also made me think it could go right.
Again, this isn’t optimism, just saying this is complex and important.