At extreme levels of empowerment you need extreme levels of having internalized human morality.
I agree that something very fraught and dangerous happens at extreme levels of empowerment. Almost no functions are safe to optimize for arbiltrarilly much (I think).
But I still claim that “human morality” isn’t a thing, and so it’s confused to say that the AI needs to have internalized human morality. I’ll probably have to write a full post about this.
And for that, I don’t see why the standard wouldn’t be “perfect human morality”. It seems to me that “basically perfect human morality” is well within our reach this or next century, if we were to be appropriately careful about how we build ASI. Like, much better value alignment than we would have gotten by just leaving it up to the evolutionary process of future generations.
I agree that something very fraught and dangerous happens at extreme levels of empowerment. Almost no functions are safe to optimize for arbiltrarilly much (I think).
But I still claim that “human morality” isn’t a thing, and so it’s confused to say that the AI needs to have internalized human morality. I’ll probably have to write a full post about this.
By this, do you mean CEV?