I think there are promising approaches to start from a benign AI initialization and optimize it to improve its performance without teaching it crazy reward-seeking behaviors. I’m working on LLM optimization techniques that hopefully maintain alignment even with an imperfect objective function. See this comment:
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI [...] Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far.
I think there are promising approaches to start from a benign AI initialization and optimize it to improve its performance without teaching it crazy reward-seeking behaviors. I’m working on LLM optimization techniques that hopefully maintain alignment even with an imperfect objective function. See this comment: