Relatedly, you implicitly equate alignment with value alignment.
No, the first 3 difficulties I explain were mainly written with sth like helpfulness/instruction-following/DWIM in mind. I think corrigibility would be an even better target for RL based AI, although I didn’t want to need to explain it in this post. I wrote:
Maybe “do what the human wants” seems simple to you? But what does this actually mean on a level that’s a bit closer to math—how might a critic evaluating this look like?
The way I think of it, “what the human wants” refers to what the human would like if they knew all the consequences of the AI’s actions. The model will surely be able to make good predictions here, but the concept seems more complex than predicting whether the human will like some text. And predicting whether the human will like some text predicts reward even better!
Maybe “follow instructions as intended” seems simple to you? Try to unpack it—how could the critic be constructed to evaluate how instruction-following a plan is, and how complex is this?
Only the last problem was specifically about value alignment, because it looks like something like CEV might be needed for an AI whose intelligence can increase arbitrarily. Or at least it’s unclear helpfulness/instruction-following would generalize if you crank up intelligence very high.
I totally agree that we currently shouldn’t aim for CEV.
No, the first 3 difficulties I explain were mainly written with sth like helpfulness/instruction-following/DWIM in mind. I think corrigibility would be an even better target for RL based AI, although I didn’t want to need to explain it in this post. I wrote:
Only the last problem was specifically about value alignment, because it looks like something like CEV might be needed for an AI whose intelligence can increase arbitrarily. Or at least it’s unclear helpfulness/instruction-following would generalize if you crank up intelligence very high.
I totally agree that we currently shouldn’t aim for CEV.