Maybe? I think there might be reasons why separation is better:
It’s more monitorable. If the RL part has to convince the character part to do stuff then it has to do so in some external channel which we can look at
If you freeze the character part, you avoid a class of failure modes related to “assistant becomes more vulnerable to persuasion” (though it’s still possible the RL part can learn to exploit existing blind spots / cognitive biases in the character part)
I don’t think it would be necessarily noticeable by reading the CoT, especially if it’s another LLM reading it. Maybe it separation is slightly better, but I don’t think it will work for AGI.
Maybe? I think there might be reasons why separation is better:
It’s more monitorable. If the RL part has to convince the character part to do stuff then it has to do so in some external channel which we can look at
If you freeze the character part, you avoid a class of failure modes related to “assistant becomes more vulnerable to persuasion” (though it’s still possible the RL part can learn to exploit existing blind spots / cognitive biases in the character part)
I don’t think it would be necessarily noticeable by reading the CoT, especially if it’s another LLM reading it. Maybe it separation is slightly better, but I don’t think it will work for AGI.