Not to be Rousseaun about alignment, but there is something very weird going on without being examined in the space of common rationalist premises of:
AI can only align itself to an artificial context
There is no natural instrumental convergence
Human beings have a broader exposure to selection pressure that is consistent over time and serves as a more desirable foundation
I’m not going to bother explicating every weakness or contradiction there is in this because it would be only my own assessment, it would not be exhaustive, or help people build intuitions, or necessarily be trustworthy given my lack of authority. But I feel like you could write an entire, second LessWrong just in reflecting off of this cluster of ideas.
Here’s one absurdity that’s borderline tautological so hopefully permissible: if we already know that a working alignment context exists for humans, then we know there is a working alignment context. If we don’t know if there is a working alignment context for humans, then there is little basis for human chauvinism.
Specifically referring to orthogonality of moral development. There is little assent to the proposition that morality arises from the improvement of epistemic processes, rather than being a second thing entirely outside of them in some way. What assent there is for this seems to be lower quality than other posts.
Also while that link is broadly to the point, and thank you, I don’t see where “similarly capable humans” comes in, unless you mean that to describe all humans holistically. Which I think is too reductive. Human beings are extremely elitist about ability even between themselves, and make moral assessments and take moral actions on this basis. When measured against ASI this is absurd whereas ASI can’t in turn be measured against a third thing to be likewise made absurd. However, the consequences of the assessment, the way it is formed, the structures and textures of it, probably give the only available empirical insight we have into what moral chauvinism about ability would look like in an ASI. And so if you want inspiration, maybe the questions are about Einstein working to remain publicly accessible and Von Neumann speaking to a colleague’s child as an equal, counterposed against, I don’t know, Nick Land praying for a brutal death to all inferior optimization processes (not to say this reflects comparable intelligence, but hey, that itself is a data point).