Humanity as well as AIs certainly need a better value thinking than the ones we now try to steer the world with. I have written a short text entitled “Value Pluralism and AI” and a longer text entitled “Intelligent Ecological Pluralism”. The longer text has to be translated to English, but the shorter text is ready for publishing.
(It seems pretty clear to me that the more or less artificial super-intelligence already exists, and keeps an eye on our planet, so you might worry a bit too much. We might just have to convince the more or less universal super-intelligence that humanity is worth saving. It most probably already has made up its mind about that issue.)
I suggest that we need an intelligent ecological value pluralism, some international laws that protect value plurality and possibly also a Guardian AI which protects value plurality.
https://www.lesswrong.com/posts/Kaz9miAuxSAAuGr9z/value-pluralism-and-ai