Saying a certain future is “not ok”, and saying gradual value drift is “business as usual”, are both value judgments. I don’t understand why you dismiss one of them but not the other, and call it “courageous and rational”.
“Business as usual” can’t be a “fallacy pattern” because it’s not a statement about facts, it’s a value statement that says we’re okay with value drift (as long as it’s some sort of “honest drift”, I assume). As you see from the others’ comments, people do really subscribe to this, so you’re not allowed to dismiss it the way you do.
it’s a value statement that says we’re okay with value drift (as long as it’s some sort of “honest drift”, I assume). And yeah, people do really subscribe to this, so you’re not allowed to dismiss it the way you do.
(Gradual) value drift → Future not optimized → Future worse than if it’s optimized → Bad. People subscribe to many crazy things. This one is incorrect pretty much by definition.
(Note that a singleton can retain optimized future while allowing different-preference or changing-preference agents to roam in its domain without disturbing the overall optimality of the setup and indeed contributing to it, if that’s preferable. This would not be an example of value drift.)
Value drift is a real-world phenomenon, something that happens to humans. No real-world phenomenon can be bad “by definition”—our morality is part of reality and cannot be affected by the sympathetic magic of dictionaries. Maybe you’re using “value drift” in a non-traditional sense, as referring to some part of your math formalism, instead of the thing that happens to humans?
No real-world phenomenon can be bad “by definition”
By definition, as in the property is logically deduced (given some necessary and not unreasonable assumptions). Consider a bucket with 20 apples of at least 100 grams each. Such buckets exist, they are “real-world phenomena”. Its weight is at least 2 kg “by definition” in the same sense as misoptimized future is worse than optimized future.
I think I figured out a way to incorporate value drift into your framework: “I value the kind of future that is well-liked by the people actually living in it, as long as they arrived at their likes and dislikes by honest value drift starting from us”. Do you think anyone making such a statement is wrong?
“I value the kind of future that is well-liked by the people actually living in it...”
If what you value happens to be what’s valued by future people, then future people are simultaneously stipulated to have the same values as you do. You don’t need the disclaimers about “honest value drift”, and there is actually no value drift.
If there is genuine value drift, then after long enough you won’t like the same situations as the future people. If you postulate that you only care about the pattern of future people liking their situation, and not other properties of that situation, you are embracing a fake simplified preference, similarly to people who claim that they only value happiness or lack of suffering.
What’s “honest value drift” and what’s good about it? Normally “value drift” makes me think of our axiology randomly losing information over time; the kind of future-with-different-values I’d value is the kind that has different instrumental values from me (because I’m not omniscient and my terminal values may not be completely consistent) but is more optimized according to my actual terminal values, presumably because that future society will have gotten better at unmuddling its terminal values, knowing enough to agree more on instrumental values, and negotiating any remaining disagreements about terminal values.
(Gradual) value drift → Future not optimized → Future worse than if it’s optimized → Bad
value drift → Future not optimized to original values → Future more aligned with new values → Bad from original viewpoint, better from new viewpoint, not optimal from either viewpoint, but what can you do?
Use of word “optimized” without specifying the value system against which the optimization took place --> variant of mind projection fallacy.
ETA: There is a very real sense in which it is axiomatic both that our value system is superior to the value system of our ancestors and that our values are superior to those of our descendants. This is not at all paradoxical—our values are better simply because they are ours, and therefore of course we see them as superior to anyone else’s values.
Where the paradox arises is in jumping from this understanding to the mistaken belief that we ought not to ever change our values.
Bad from original viewpoint, better from new viewpoint, not optimal from either viewpoint, but what can you do?
Where the paradox arises is in jumping from this understanding to the mistaken belief that we ought not to ever change our values.
A compelling moral argument may change our values, but not our moral frame of reference.
The moral frame of reference is like a forking bush of possible future value systems stemming from a current human morality; it represents human morality’s ability to modify itself upon hearing moral arguments.
The notion of moral argument and moral progress is meaningful within my moral frame of reference, but not meaningful relative to a paperclipper utility function. A paperclipper will not ever switch to stapler maximization on any moral argument; a consistent paperclipper does not think that it will possibly modify its utility function upon acquiring new information. In contrast, I think that I will possibly modify my morality for the better, it’s just that I don’t yet know the argument that will compel me, because if I knew it I would have already changed my mind.
It is not impossible that paperclipping is the endpoint to all moral progress, and there exists a perfectly compelling chain of reasoning that converts all humans to paperclippers. It is “just” vanishingly unlikely. We cannot, of course, observe our moral frame of reference from an outside omniscient vantage point but we’re able to muse about it.
If we do assume omniscience for a second, then there is a space of values that humans would never willingly modify themselves into. Value drift means drifting into such space rather than a modification of values in general.
There is a very real sense in which it is axiomatic both that our value system is superior to the value system of our ancestors and that our values are superior to those of our descendants. This is not at all paradoxical—our values are better simply because they are ours, and therefore of course we see them as superior to anyone else’s values.
If our ancestors and our descendant are in the same moral frame of reference then you could possibly convert your ancestors or most of your ancestors to your morality and be converted to future morality by future people. Of course it is not easy to say which means of conversion are valid; on the most basic level I’d say that rearranging your brains’ atoms to a paperclipper breaks out of the frame of reference while verbal education and arguments generally don’t.
If we do assume omniscience for a second, then there is a space of values that humans would never willingly modify themselves into. Value drift means drifting into such space rather than a modification of values in general.
Rather (in your terminology), value drift is change in the moral frame of reference, even if (current instrumental) morality stays the same.
I agree, it seems a more general way of putting it.
Anyway, now that you mention it I’m intrigued and slightly freaked out by a scenario in which my frame of reference changes without my current values changing. First, is it even knowable when it happens? All our reasoning is based on current values. If an alien race comes and modifies us in a way that our future moral progress changes but not our current values, we could never know the change happened at all. It is a type of value loss that preserves reflective consistency. I mean, we wouldn’t agree to be changed to paperclippers but on what basis could we refuse an unspecified change to our moral frame of reference (leaving current values intact)?
I’m not sure I understand this talk of “moral frames of reference” vs simply “values”.
But would an analogy to frame change be theory change? As when we replace Newton’s theory of gravity with Einstein’s theory, leaving the vast majority of theoretical predictions intact?
In this analogy, we might make the change (in theory or moral frame) because we encounter new information (new astronomical or moral facts) that impel the change. Or, we might change for the same reason we might change from the Copenhagen interpretation to MWI—it seems to work just as well, but has greater elegance.
I’m not sure I understand this talk of “moral frames of reference” vs simply “values”.
By analogy, take a complicated program as “frame of reference”, and state of knowledge about what it outputs “current values”. As you learn more, “current values” change, but frame of reference, defining the subject matter, stays the same and determines the direction of discovering more precise “current values”.
Note that the exact output may well be unknowable in its explicit form, but “frame of reference” says precisely what it is. Compare with infinite mathematical structures that can never be seen “explicitly”, but with the laws of correct reasoning about them perfectly defined.
As you learn more, “current values” change, but frame of reference, defining the subject matter, stays the same and determines the direction of discovering more precise “current values”.
Is there potential divergence of “current values” in this analogy (or in your model of morality)?
Moral frame of reference determines the direction in exploration of values, but you can’t explicitly know this direction, even if you know its definition, because otherwise you’d already be there. It’s like with definition of action in ambient control. When definition is changed, you have no reason to expect that the defined thing remains the same, even though at that very moment your state of knowledge about the previous definition might happen to coincide with your state of knowledge about the new definition. And then the states of knowledge go their different ways.
Saying a certain future is “not ok”, and saying gradual value drift is “business as usual”, are both value judgments. I don’t understand why you dismiss one of them but not the other, and call it “courageous and rational”.
I don’t understand your comment enough to reply (“business as usual”? - that was more of a fallacy pattern). Maybe if you write in more detail?
“Business as usual” can’t be a “fallacy pattern” because it’s not a statement about facts, it’s a value statement that says we’re okay with value drift (as long as it’s some sort of “honest drift”, I assume). As you see from the others’ comments, people do really subscribe to this, so you’re not allowed to dismiss it the way you do.
(Gradual) value drift → Future not optimized → Future worse than if it’s optimized → Bad. People subscribe to many crazy things. This one is incorrect pretty much by definition.
(Note that a singleton can retain optimized future while allowing different-preference or changing-preference agents to roam in its domain without disturbing the overall optimality of the setup and indeed contributing to it, if that’s preferable. This would not be an example of value drift.)
Value drift is a real-world phenomenon, something that happens to humans. No real-world phenomenon can be bad “by definition”—our morality is part of reality and cannot be affected by the sympathetic magic of dictionaries. Maybe you’re using “value drift” in a non-traditional sense, as referring to some part of your math formalism, instead of the thing that happens to humans?
By definition, as in the property is logically deduced (given some necessary and not unreasonable assumptions). Consider a bucket with 20 apples of at least 100 grams each. Such buckets exist, they are “real-world phenomena”. Its weight is at least 2 kg “by definition” in the same sense as misoptimized future is worse than optimized future.
I think I figured out a way to incorporate value drift into your framework: “I value the kind of future that is well-liked by the people actually living in it, as long as they arrived at their likes and dislikes by honest value drift starting from us”. Do you think anyone making such a statement is wrong?
It’s a start. But note that it is trivially satisfied by a world that has no one living in it.
If what you value happens to be what’s valued by future people, then future people are simultaneously stipulated to have the same values as you do. You don’t need the disclaimers about “honest value drift”, and there is actually no value drift.
If there is genuine value drift, then after long enough you won’t like the same situations as the future people. If you postulate that you only care about the pattern of future people liking their situation, and not other properties of that situation, you are embracing a fake simplified preference, similarly to people who claim that they only value happiness or lack of suffering.
What’s “honest value drift” and what’s good about it? Normally “value drift” makes me think of our axiology randomly losing information over time; the kind of future-with-different-values I’d value is the kind that has different instrumental values from me (because I’m not omniscient and my terminal values may not be completely consistent) but is more optimized according to my actual terminal values, presumably because that future society will have gotten better at unmuddling its terminal values, knowing enough to agree more on instrumental values, and negotiating any remaining disagreements about terminal values.
value drift → Future not optimized to original values → Future more aligned with new values → Bad from original viewpoint, better from new viewpoint, not optimal from either viewpoint, but what can you do?
Use of word “optimized” without specifying the value system against which the optimization took place --> variant of mind projection fallacy.
ETA: There is a very real sense in which it is axiomatic both that our value system is superior to the value system of our ancestors and that our values are superior to those of our descendants. This is not at all paradoxical—our values are better simply because they are ours, and therefore of course we see them as superior to anyone else’s values.
Where the paradox arises is in jumping from this understanding to the mistaken belief that we ought not to ever change our values.
A compelling moral argument may change our values, but not our moral frame of reference.
The moral frame of reference is like a forking bush of possible future value systems stemming from a current human morality; it represents human morality’s ability to modify itself upon hearing moral arguments.
The notion of moral argument and moral progress is meaningful within my moral frame of reference, but not meaningful relative to a paperclipper utility function. A paperclipper will not ever switch to stapler maximization on any moral argument; a consistent paperclipper does not think that it will possibly modify its utility function upon acquiring new information. In contrast, I think that I will possibly modify my morality for the better, it’s just that I don’t yet know the argument that will compel me, because if I knew it I would have already changed my mind.
It is not impossible that paperclipping is the endpoint to all moral progress, and there exists a perfectly compelling chain of reasoning that converts all humans to paperclippers. It is “just” vanishingly unlikely. We cannot, of course, observe our moral frame of reference from an outside omniscient vantage point but we’re able to muse about it.
If we do assume omniscience for a second, then there is a space of values that humans would never willingly modify themselves into. Value drift means drifting into such space rather than a modification of values in general.
If our ancestors and our descendant are in the same moral frame of reference then you could possibly convert your ancestors or most of your ancestors to your morality and be converted to future morality by future people. Of course it is not easy to say which means of conversion are valid; on the most basic level I’d say that rearranging your brains’ atoms to a paperclipper breaks out of the frame of reference while verbal education and arguments generally don’t.
Rather (in your terminology), value drift is change in the moral frame of reference, even if (current instrumental) morality stays the same.
I agree, it seems a more general way of putting it.
Anyway, now that you mention it I’m intrigued and slightly freaked out by a scenario in which my frame of reference changes without my current values changing. First, is it even knowable when it happens? All our reasoning is based on current values. If an alien race comes and modifies us in a way that our future moral progress changes but not our current values, we could never know the change happened at all. It is a type of value loss that preserves reflective consistency. I mean, we wouldn’t agree to be changed to paperclippers but on what basis could we refuse an unspecified change to our moral frame of reference (leaving current values intact)?
I’m not sure I understand this talk of “moral frames of reference” vs simply “values”.
But would an analogy to frame change be theory change? As when we replace Newton’s theory of gravity with Einstein’s theory, leaving the vast majority of theoretical predictions intact?
In this analogy, we might make the change (in theory or moral frame) because we encounter new information (new astronomical or moral facts) that impel the change. Or, we might change for the same reason we might change from the Copenhagen interpretation to MWI—it seems to work just as well, but has greater elegance.
By analogy, take a complicated program as “frame of reference”, and state of knowledge about what it outputs “current values”. As you learn more, “current values” change, but frame of reference, defining the subject matter, stays the same and determines the direction of discovering more precise “current values”.
Note that the exact output may well be unknowable in its explicit form, but “frame of reference” says precisely what it is. Compare with infinite mathematical structures that can never be seen “explicitly”, but with the laws of correct reasoning about them perfectly defined.
Is there potential divergence of “current values” in this analogy (or in your model of morality)?
Moral frame of reference determines the direction in exploration of values, but you can’t explicitly know this direction, even if you know its definition, because otherwise you’d already be there. It’s like with definition of action in ambient control. When definition is changed, you have no reason to expect that the defined thing remains the same, even though at that very moment your state of knowledge about the previous definition might happen to coincide with your state of knowledge about the new definition. And then the states of knowledge go their different ways.