The upshot is that (afaict) there’s no such thing in humans as “working out one’s true values” apart from an environment, where for humans the most salient feature of the environment (for boring EvoPsych reasons) is what the people around one are like and how they’ll react. People who think they’re “working out their true values” in the sense of crystalizing facts about themselves, rather than running forward a state-function of the the self, friends, and environment, are (on this view) just self-deceiving.
How do you think Jeremy Bentham came to the conclusion that animal welfare matters morally and that there’s nothing morally wrong with homosexuality? Are you claiming that he ran forward a computation of how the relevant parts of his social milieu are going to react, and did what maximized the expected value of reaction?
I buy that this is how most of human “value formation” happens, but I don’t buy that this is all that happens. I think that humans vary in some trait similar to the need for cognition (probably positively correlated), which is something like “how much one is bothered by one’s value dissonances”, independent of social surroundings.
Like, you could tell a similar history about intellectual/scientific/technological progress, and it would be directionally right, but not entirely right, and the “not entirely” matters a lot.
Aside from all that, I expect that a major part of AIs’ equivalent of social interaction will be other AIs or general readouts of things on the internet downstream of human and non-human activity that do not exert a strong pressure in the direction of being more human-friendly, especially given that AIs do not share the human social machinery (as Ray says).
How do you think Jeremy Bentham came to the conclusion that animal welfare matters morally and that there’s nothing morally wrong with homosexuality? Are you claiming that he ran forward a computation of how the relevant parts of his social milieu are going to react, and did what maximized the expected value of reaction?
I buy that this is how most of human “value formation” happens, but I don’t buy that this is all that happens. I think that humans vary in some trait similar to the need for cognition (probably positively correlated), which is something like “how much one is bothered by one’s value dissonances”, independent of social surroundings.
Like, you could tell a similar history about intellectual/scientific/technological progress, and it would be directionally right, but not entirely right, and the “not entirely” matters a lot.
Aside from all that, I expect that a major part of AIs’ equivalent of social interaction will be other AIs or general readouts of things on the internet downstream of human and non-human activity that do not exert a strong pressure in the direction of being more human-friendly, especially given that AIs do not share the human social machinery (as Ray says).