I don’t expect anyone to change their values. I think the average human’s values, applied with great power and intelligence, would produce good things for people (and animals and sentient AIs).
That’s because I think most people have more goodwill than ill-will toward sentient beings. They may harbor grudges toward some particular people or types of people/minds, but the average will still probably be really good on.
They don’t have to care about correctness, completeness, or utility-maximization a bit. They just have tell their ASI to make life better for people, in as much or little detail as they want. If they care about people’s wellbeing even a tiny bit more than they want them to suffer, this seems very likely to happen.
That doesn’t make me want to create obedient ASI. We could get a bad draw of someone who’s just plain sadistic toward most sentients. Based on my readings on malevolent people (sociopathy etc) and other psychological studies, I think that’s actually less than 1% of humanity—maybe much less. But sociopaths/malevolents are overrepresented in positions of power. So I don’t think this is anything like a safe bet EVEN IF I’m right that most people feel more empathy than sadism toward most beings.
Which I’m not sure of. You say “isn’t necessarily” and “not guaranteed” and frame it as disagreement, but I agree. I used those same qualifiers in my piece—pretty heavily I think. I liked the comment “Seth seems unsure of a lot of stuff” because I am, and I want to convey that as a central point. I think everyone should feel unsure. What people would do with unlimited power or unlimited knowledge has rarely even been thought about, let alone analyzed carefully. Historical analyses are all about what people will do with the relative tiny scraps of power and knowledge in history or most thought experiments. ASI changes the situation dramatically, in ways we just haven’t thought through much at all.
Thanks for the careful response! I appreciate you reading the longer version.
Your response gave me the above idea for stating the core logic more clearly.
I don’t expect anyone to change their values. I think the average human’s values, applied with great power and intelligence, would produce good things for people (and animals and sentient AIs).
That’s because I think most people have more goodwill than ill-will toward sentient beings. They may harbor grudges toward some particular people or types of people/minds, but the average will still probably be really good on.
They don’t have to care about correctness, completeness, or utility-maximization a bit. They just have tell their ASI to make life better for people, in as much or little detail as they want. If they care about people’s wellbeing even a tiny bit more than they want them to suffer, this seems very likely to happen.
That doesn’t make me want to create obedient ASI. We could get a bad draw of someone who’s just plain sadistic toward most sentients. Based on my readings on malevolent people (sociopathy etc) and other psychological studies, I think that’s actually less than 1% of humanity—maybe much less. But sociopaths/malevolents are overrepresented in positions of power. So I don’t think this is anything like a safe bet EVEN IF I’m right that most people feel more empathy than sadism toward most beings.
Which I’m not sure of. You say “isn’t necessarily” and “not guaranteed” and frame it as disagreement, but I agree. I used those same qualifiers in my piece—pretty heavily I think. I liked the comment “Seth seems unsure of a lot of stuff” because I am, and I want to convey that as a central point. I think everyone should feel unsure. What people would do with unlimited power or unlimited knowledge has rarely even been thought about, let alone analyzed carefully. Historical analyses are all about what people will do with the relative tiny scraps of power and knowledge in history or most thought experiments. ASI changes the situation dramatically, in ways we just haven’t thought through much at all.
Thanks for the careful response! I appreciate you reading the longer version.
Your response gave me the above idea for stating the core logic more clearly.