I’d suggest TurnTrout’s writing (Alex Turner at DeepMind), since he’s the person who first came up with the idea. Most of his posts are on LessWrong/The Aligment Forum, but they’re best organized on his own website. I’d suggest starting at https://turntrout.com/research, reading the section on Shard Theory, and following links.
He himself admits that some of his key posts often seem to get misunderstood: I think they repay careful reading and some thought.
I’ve though about this some more and I think what you mean (leaving aside physical and homeostatic values and focusing on organism-wide values) is that, even if we define our “terminal value” as I have above, whence the basket of goods that mean “happiness/flourishing” to me?
After thinking yet more about this, I realize that the rock bottom terminal value I am trying to identify isn’t the basket of goods itself, but my valuing of it. This seems to be a meta-value. “Valuing” itself.
If I were seconds away from dying of thirst, I might sell many terminally valuable goods for water. But if to get water I had to give up terminally valuing...I’m not sure I’d want to bother with the water or staying alive.
Maybe this meta-value comes from evolution too...except that, would that mean that it’s possible we could have not evolved it, and still been sentient beings? Because that is hard to imagine.
I guess what I’m saying is that the terminal value is not the basket...it is for the basket. Meaning that the rock-bottom is dynamic desiring. No particular value is static.
I’d suggest TurnTrout’s writing (Alex Turner at DeepMind), since he’s the person who first came up with the idea. Most of his posts are on LessWrong/The Aligment Forum, but they’re best organized on his own website. I’d suggest starting at https://turntrout.com/research, reading the section on Shard Theory, and following links.
He himself admits that some of his key posts often seem to get misunderstood: I think they repay careful reading and some thought.
After thinking yet more about this, I realize that the rock bottom terminal value I am trying to identify isn’t the basket of goods itself, but my valuing of it. This seems to be a meta-value. “Valuing” itself.
If I were seconds away from dying of thirst, I might sell many terminally valuable goods for water. But if to get water I had to give up terminally valuing...I’m not sure I’d want to bother with the water or staying alive.
Maybe this meta-value comes from evolution too...except that, would that mean that it’s possible we could have not evolved it, and still been sentient beings? Because that is hard to imagine.
I guess what I’m saying is that the terminal value is not the basket...it is for the basket. Meaning that the rock-bottom is dynamic desiring. No particular value is static.