You’re right in a sense that we’d like to avoid it, but if it occurs gradually, it feels much more like “we just changed our minds” (like we definitely don’t value “honor” as much as the ancient greeks, etc), as compared to “we and our values were wiped out”.
The problem is not with “losing our values”, it’s about the future being optimized to something other than our values. The details of the process that leads to the incorrectly optimized future are immaterial, it’s the outcome that matters. When I say “our values”, I’m referring to a fixed idea, which doesn’t depend on what happens in the future, in particular it doesn’t depend on whether there are people with these or different values in the future.
I think one reason why people (including me, in the past) have difficulty accepting the way you present this argument is that you’re speaking in too abstract terms, while many of the values that we’d actually like to preserve are ones that we appreciate the most if we consider them in “near” mode. It might work better if you gave concrete examples of ways by which there could be a catastrophic value drift, like naming Bostrom’s all-work-and-no-fun scenario where
what will maximize fitness in the future will be nothing but non-stop high-intensity drudgery, work of a drab and repetitive nature, aimed at improving the eighth decimal of some economic output measure
You’re right in a sense that we’d like to avoid it, but if it occurs gradually, it feels much more like “we just changed our minds” (like we definitely don’t value “honor” as much as the ancient greeks, etc), as compared to “we and our values were wiped out”.
The problem is not with “losing our values”, it’s about the future being optimized to something other than our values. The details of the process that leads to the incorrectly optimized future are immaterial, it’s the outcome that matters. When I say “our values”, I’m referring to a fixed idea, which doesn’t depend on what happens in the future, in particular it doesn’t depend on whether there are people with these or different values in the future.
I think one reason why people (including me, in the past) have difficulty accepting the way you present this argument is that you’re speaking in too abstract terms, while many of the values that we’d actually like to preserve are ones that we appreciate the most if we consider them in “near” mode. It might work better if you gave concrete examples of ways by which there could be a catastrophic value drift, like naming Bostrom’s all-work-and-no-fun scenario where
or some similar example.