I think this post is counterproductive. There are serious reasons to believe why iterative alignment would fail, and serious reasons to believe that it’s the best thing we can work on right now. But this post reads like 30% vague ideas and 70% condescension. It feels like it’s written to score social points rather than put forth good ideas in earnest discussion.
The idea explained in the post, in a way that I don’t know what other reference already explains, is that there is a disconnect between the expected character of a mature goal engine, and the nature of the tools that are being developed under the name “AI safety”.
I think this post is counterproductive. There are serious reasons to believe why iterative alignment would fail, and serious reasons to believe that it’s the best thing we can work on right now. But this post reads like 30% vague ideas and 70% condescension. It feels like it’s written to score social points rather than put forth good ideas in earnest discussion.
The idea explained in the post, in a way that I don’t know what other reference already explains, is that there is a disconnect between the expected character of a mature goal engine, and the nature of the tools that are being developed under the name “AI safety”.