Edited to clarify that this isn’t what I’m saying. Added:
If there’s simply a tradeoff between them, we might still want to sacrifice accurate beliefs and effective action for happiness. But what I’m gesturing towards is the idea that happiness might not actually be a concept which makes much sense given a complete understanding of minds—as implied by the buddhist view of happiness as an illusion, for example.
But it’s not like happiness is the Tooth Fairy. It’s an honest ingredient in useful models I have of human beings, at some level of abstraction. If you think future decision-makers might decide happiness “isn’t real,” what I hear is that they’re ditching those models of humans that include happiness, and deciding to use models that never mention it or use it.
And I would consider this a fairly straightforward failure to align with my meta-preferences (my preferences about what should count as my preferences). I don’t want to be modeled in ways that don’t include happiness, and this is a crucial part of talking about my preferences, because there is no such thing as human preferences divorced from any model of the world used to represent them.
I agree that some people are imagining that we’ll end up representing human values in some implicitly-determined, or “objective” model of the world. And such a process might end up not thinking happiness is a good model ingredient. To me, this sounds like a solid argument to not do that.
Edited to clarify that this isn’t what I’m saying. Added:
Alright.
But it’s not like happiness is the Tooth Fairy. It’s an honest ingredient in useful models I have of human beings, at some level of abstraction. If you think future decision-makers might decide happiness “isn’t real,” what I hear is that they’re ditching those models of humans that include happiness, and deciding to use models that never mention it or use it.
And I would consider this a fairly straightforward failure to align with my meta-preferences (my preferences about what should count as my preferences). I don’t want to be modeled in ways that don’t include happiness, and this is a crucial part of talking about my preferences, because there is no such thing as human preferences divorced from any model of the world used to represent them.
I agree that some people are imagining that we’ll end up representing human values in some implicitly-determined, or “objective” model of the world. And such a process might end up not thinking happiness is a good model ingredient. To me, this sounds like a solid argument to not do that.