This basically boils down to the alignment problem. We don’t know how to specify what we want, but that doesn’t mean it is necessarily incoherent.
Treating the human skull as “some kind of unviolable and uninfluenceable black box” seems to get you some of the way there, but of course is problematic in its own ways (e.g. you wouldn’t want delusional AIs). Still it seems like it points to the path forwards in a way.
This basically boils down to the alignment problem. We don’t know how to specify what we want, but that doesn’t mean it is necessarily incoherent.
Treating the human skull as “some kind of unviolable and uninfluenceable black box” seems to get you some of the way there, but of course is problematic in its own ways (e.g. you wouldn’t want delusional AIs). Still it seems like it points to the path forwards in a way.