My model of how human values arrive naturally from how the human brain makes sense of the world (all of the below steps can happen subconsciously):
First, the brain learns how the body functions (details left as an exercise for the reader). The brain then figures out that the body works better if senses and reflexes are coordinated. Noticing patterns and successful movement and action feels good.
Then the brain discovers the abstraction of interests and desires and that the body works better (gets the nutrients and rest that it needs) if interests and desires are followed. Following your wants feels rewarding.
Then the brain notices personal relationships and that interests and wants are better satisfied if relationships are cultivated (the win-win from cooperation). Having a good relationship feels good, and the thought of the loss of a relationship feels painful.
The brain then discovers the commonalities of expectations within groups—group norms and values—and that relationships are easier to maintain and have less conflict if a stable and predictable identity is presented to other people. Adhering to group norms and having stable values feels rewarding.
I think values aren’t the end of it. Kegan’s stages of adult development have a further stage where the brain learns to deal with inter-group tensions by being more fluid. I think this relaxes constraints and smoothes the brain’s model and roughly corresponds to what Aging Well calls Integrity or what Paul Graham calls Keep Your Identity Small. And there may be consolidation beyond that—who knows what an AGI would pick up.
My model of how human values arrive naturally from how the human brain makes sense of the world (all of the below steps can happen subconsciously):
First, the brain learns how the body functions (details left as an exercise for the reader). The brain then figures out that the body works better if senses and reflexes are coordinated. Noticing patterns and successful movement and action feels good.
Then the brain discovers the abstraction of interests and desires and that the body works better (gets the nutrients and rest that it needs) if interests and desires are followed. Following your wants feels rewarding.
Then the brain notices personal relationships and that interests and wants are better satisfied if relationships are cultivated (the win-win from cooperation). Having a good relationship feels good, and the thought of the loss of a relationship feels painful.
The brain then discovers the commonalities of expectations within groups—group norms and values—and that relationships are easier to maintain and have less conflict if a stable and predictable identity is presented to other people. Adhering to group norms and having stable values feels rewarding.
I think values aren’t the end of it. Kegan’s stages of adult development have a further stage where the brain learns to deal with inter-group tensions by being more fluid. I think this relaxes constraints and smoothes the brain’s model and roughly corresponds to what Aging Well calls Integrity or what Paul Graham calls Keep Your Identity Small. And there may be consolidation beyond that—who knows what an AGI would pick up.