I like the idea that rather than trying to impress a judge, the AI simply learns what a more knowledgeable version of itself would say, as specified by an interpretable hint. This hint could be written by the AI itself, which means that unlike in RL (which eventually forces the AI to monomaniacally chase reward), the AI can use its own discretion to improve its own cognitive patterns while maintaining alignment. It reminds me of this comment I wrote a year ago:
Ideally, we could do pure SFT to specify a “really nice guy,” then let that guy reflect deeply about how to improve himself. Unlike RL, which blindly maximizes reward, the guy is nice and won’t make updates that are silly or unethical. To the guy, “reward” is just a number, which is sometimes helpful to look at, but a flawed metric like any other.
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI, like in (my interpretation of) OPSD.
Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far. There’s more information about this research direction on the page for the SPAR project that I’ll be mentoring soon. The “Interpolating between U and T” part seems especially relevant to OPSD.
I wonder if these ideas could be combined: limiting the length or number of hints used in OPSD updates, or limiting the divergence of the OPSD distribution to distill from (seems similar to the DemoPSD paper you linked).
The OPSD stuff is very interesting!
I like the idea that rather than trying to impress a judge, the AI simply learns what a more knowledgeable version of itself would say, as specified by an interpretable hint. This hint could be written by the AI itself, which means that unlike in RL (which eventually forces the AI to monomaniacally chase reward), the AI can use its own discretion to improve its own cognitive patterns while maintaining alignment. It reminds me of this comment I wrote a year ago:
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI, like in (my interpretation of) OPSD.
Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far. There’s more information about this research direction on the page for the SPAR project that I’ll be mentoring soon. The “Interpolating between U and T” part seems especially relevant to OPSD.
I wonder if these ideas could be combined: limiting the length or number of hints used in OPSD updates, or limiting the divergence of the OPSD distribution to distill from (seems similar to the DemoPSD paper you linked).