Optimizing a mind for competence also optimizes it for coherence, because incoherence is generally detrimental to competence. The longer and more complicated a task, the more true this seems to be. I predict alignment research to be an unusually long and complex task.
I would not call it necessary, but current methods do not really allow us to screen for one thing and not the other. So we’ll get both.
Why do we think that reflection is neccesary for competence. That is, competence does not seem to imply coherence, unless I missed an argument.
Optimizing a mind for competence also optimizes it for coherence, because incoherence is generally detrimental to competence. The longer and more complicated a task, the more true this seems to be. I predict alignment research to be an unusually long and complex task.
I would not call it necessary, but current methods do not really allow us to screen for one thing and not the other. So we’ll get both.