I think your first point basically covers why—people are worried about alignment difficulties in superhuman systems, in particular (because those are the dangerous systems which can cause existential failures). I think a lot of current RLHF work is focused on providing reward signals to current systems in ways that don’t directly address the problem of “how do we reward systems with behaviors that have consequences that are too complicated for humans to understand”.
I think your first point basically covers why—people are worried about alignment difficulties in superhuman systems, in particular (because those are the dangerous systems which can cause existential failures). I think a lot of current RLHF work is focused on providing reward signals to current systems in ways that don’t directly address the problem of “how do we reward systems with behaviors that have consequences that are too complicated for humans to understand”.