“concrete candidates for what the value system should be”
I have actually created at least one post where I tried to explain what it could be. IMO the value system should[1] be either mentors who teach humans something that others have already discovered and don’t keep secret, amplify human capabilities and have the human do most challenging aspects of the work or protectors who prevent mankind as a whole from being destroyed, but not servants who do all the work. The latter option, unlike the other two, leads to the Deep Utopia or the scenarios where the oligarchs stop needing to have humans.
“concrete candidates for what the value system should be”
I have actually created at least one post where I tried to explain what it could be. IMO the value system should[1] be either mentors who teach humans something that others have already discovered and don’t keep secret, amplify human capabilities and have the human do most challenging aspects of the work or protectors who prevent mankind as a whole from being destroyed, but not servants who do all the work. The latter option, unlike the other two, leads to the Deep Utopia or the scenarios where the oligarchs stop needing to have humans.
A more radical version of my argument is that any AI aligns itself either to my proposal or to the AI takeover.