When we think we want to go to the Chinese food restaurant, it’s because we want to eat the food. When we learn that it’s closed and don’t wanna go anymore, that’s not “value drift” to hold off, it’s learning that what we thought we wanted isn’t serving what we really want.
When we think we want to eat the Chinese food, it’s not because of some terminal “It’s just really good”. Get food poisoning once, and you’ll notice that there is indeed accounting for taste. “Yummy” becomes “Yucky” when we realize that the underlying value isn’t being delivered. I’ve even seen things as “innate and fundamental” as sexual orientation shift as it becomes clear that there is/isn’t value to be delivered.
How can you prevent the drift of value, but by sticking one’s head in the sand and refusing to learn that it’s not giving us what we want? How can we want to prevent this “value drift”, once we “know more and think faster” enough to understand the mechanism? What you describe as “getting the self-control first” is making sure to not get the knowledge.
“Realizing things” is not separate from “self-modification”, such that they can come in different orders. Realizing things is what modifies the self. Realizing that the Chinese restaurant is closed modifies your self. Realizing that your spouse has been cheating on you modifies your self. Realizing that Jews aren’t actually what you need them to be changes your self.
The Hitler who updates on what his Jew hatred brought him becomes a different Hitler, regardless of the order, so long as he does the update. Some orders may make it easier to accept the update than others, and therefore might make it more or less likely that he updates, but if he learns that the value is not delivered then he learns the value is not delivered. And if it’s true, and he doesn’t learn, then he has not CE’ed his V.
I agree that the process of learning always changes you. Tasting a food you have never tried before switches you from neutral-ish[1] to either liking the food or disliking the food; either one is a change.
This process is not linear. For example, there is food (coffee, blue cheese, beer, dry wine...) that tastes terrible on first try to most people, but if you stick with it, sometimes you reprogram your brain to start enjoying it. If you are already reprogrammed, the choice is already made, but if you are a beginner… does it make more sense to respect your current preferences and avoid the food, or to keep eating it anyway because you know this decision will make you happy after you reprogram your brain?
I guess in real life, many people simply get reprogrammed under social pressure. Stage one, they would prefer not to do it, but they have a stronger preference to be perceived as cool by their environment. Stage two, now they are reprogrammed and there is no conflict.
But suppose that you are perfectly above all social pressure, and you happen to not like beer at the moment. Is there a good reason for you to choose the reprogramming anyway? Sure, if you do, you will be happy that you did. But if you won’t, you won’t miss it. (Maybe curiosity is the remaining argument for the change? But without the social pressure, there is no reason to be curious about this specific thing.) So it seems to me that there are two different reflectively stable outcomes, and it depends on history where you end up.
We could even go further; with superior self-modification skills I could give myself a completely arbitrary preference, for example a sexual fetish for triangles. It seems silly now, but after modifying myself, and decorating my palace with paintings of triangles, I would probably be happy that I did it. I may even feel a little curious right now about what such absurd situation would be like. So should I modify myself that way? Seems like the answer is no, but maybe if I had too much time and got bored… well, I also don’t have a strong reason against doing that, except for some general “let’s not do arbitrary self-modifications, even if individually mostly harmless, the cumulative effect might dilute too much the things I care about now”.
Perhaps good and evil[2] are also acquired tastes; I am not sure about this part, but it feels plausible.
“Realizing things” is not separate from “self-modification”, such that they can come in different orders. Realizing things is what modifies the self.
All new knowledge is self-modification, but not all self-modification is new (external) knowledge. You can also self-modify by resolving internal tensions. Or by changing a random connection in your brain. Also, if you obtain new knowledge in different order, the later information gets interpreted in the light of the former.
The all-knowing Hitler would know that his original reasons for hating Jews are no longer valid, but he might retain an aesthetic preference for doing so, for example because the very emotion of feeling superior to someone feels enjoyable.
I am not going to provide an exact definition, but I think there is some generic desire for the world to be a nice and happy place also for others, versus just not caring about that and therefore effectively sacrificing the happiness of others for whatever things I do care about.
But suppose that you are perfectly above all social pressure, and you happen to not like beer at the moment. Is there a good reason for you to choose the reprogramming anyway? Sure, if you do, you will be happy that you did. But if you won’t, you won’t miss it. (Maybe curiosity is the remaining argument for the change? But without the social pressure, there is no reason to be curious about this specific thing.) So it seems to me that there are two different reflectively stable outcomes, and it depends on history where you end up.
In practice, this path dependency thing is indeed important, but it has a lot to do with our tendency to get lost and fail to find coherence.
For example, instead of beer what about heroin? Needles are icky, but boy will that change if you give it a shot! What happens in the longer term isn’t so simple as “more happy” though, and especially when the effects come from exogenous chemicals, we can’t really trust our initial pleasure to cache out in anything real.
This can make heroin very risky because people will often fail to learn that heroin injections are yucky again, but that is where the road coheres to. I don’t have any experience with heroin, but I have tried legally prescribed opioids a couple times and went through this arc. After the first time I couldn’t stop thinking about it for a month because it felt so good. Eventually though, my brain kinda recognized that this is not actually a good thing, I don’t actually want it, and when I eventually tried it again “just because” it wasn’t even enjoyable.
Beer and coffee are a lot more subtle and have context dependent social stuff going on, but “If you do you’ll be glad you did” is far from a sure thing.
We could even go further; with superior self-modification skills I could give myself a completely arbitrary preference, for example a sexual fetish for triangles. It seems silly now, but after modifying myself, and decorating my palace with paintings of triangles, I would probably be happy that I did it. I may even feel a little curious right now about what such absurd situation would be like. So should I modify myself that way?
I mean… I kinda vote “yes”. Because curiosity is important, the things you learn from experience are important, and this is a relatively harmless example to experiment with.
In my experience though, arbitrary modifications like this aren’t very stable. If there isn’t any actual value being delivered, and you don’t try to set up a labyrinth of motivations to not-look, people tend to learn that triangles aren’t so exciting as they had tricked themselves into believing.
All new knowledge is self-modification, but not all self-modification is new (external) knowledge. You can also self-modify by resolving internal tensions. Or by changing a random connection in your brain.
A railroad spike to the brain will indeed indeed modify you, and is not well described as “learning new things!”. But resolving internal tensions usually is. And the railroad spike to the brain/random connection change is generally well described as losing things you have learned, or learning (expected) falsehoods.
Also, if you obtain new knowledge in different order, the later information gets interpreted in the light of the former.
This is another “probably in practice, but only because we don’t reach coherence” things. Maybe you consider the latter in terms of the former without going back to reinterpret the former in terms of the latter, but Bayes doesn’t justify this failure to propagate updates with any sort of path dependence.
The all-knowing Hitler would know that his original reasons for hating Jews are no longer valid, but he might retain an aesthetic preference for doing so, for example because the very emotion of feeling superior to someone feels enjoyable.
My point is that what we think of as inscrutable aesthetic preferences are built upon implicit beliefs about the world, and updating the underlying structure changes the aesthetics. Hitler may like to feel superior, but his potential superiority is itself a fact about reality that he could update on.
What happens when you sit with the question “Are you superior?”?
There’s often an impulse to flinch away, and refuse to update, but when you do, things change.
Then how does one tell apant the true terminal values and instrumental ones? Does it mean that the CEV of an individual human is likely to be some combo of satisfaction of primitive values, fun-theoreticones, idiosyncratic ones and of a way to instill decision-theoretic results (like coordinating with others in prisoner-like dilemmas) into our primitive brains? And how would the latter two value types be changeable? How would they change in AIs?
Then how does one tell apant the true terminal values and instrumental ones?
I don’t know.
So like, one time I was playing football with my cousins on Thanksgiving, and hurt my foot bad enough that I thought it was broken. As I dug into why reality was different than I wanted, at first there was always something underneath.
“I wish my foot wasn’t hurting. Of course I wish my foot wasn’t hurting, who wouldn’t? I acknowledge the fact that my foot is hurting. Why is it hurting? Because it’s broken, lol.”
So then “I wish my foot wasn’t broken”, why’s it broken? Oh, because I was playing football and tripped. Why’d I trip? Shit just happens, man. So, “Of course shit happens, so of course I broke my foot when I was playing football with my family, so of course it hurts.” What’s the problem? Nothing, actually. Suffering resolved.
In that case, my desire to not be in pain bottomed out at not wanting to have hurt myself unnecessarily, and “shit just happens”, but like… the fact that it was serving as a terminal in this context is still dependent on the fact that I didn’t believe I could do anything about it. If there had been some nootropic that cuts the rate of mistakes to 10%, then “Why did shit happen?” now has an answer of “Because you didn’t improve your brain function, dummy”, and we’re back to the races.
I know how to get to the practical bottom of things, for a given context. But as some sort of general case where we remove “all” practical limits… I dunno man. I’m not sure the question is well formed. I’m not sure it’s not.
Does it mean that the CEV of an individual human
I don’t know where it grounds out, just where it doesn’t. Which is useful on the margin, and maybe even large margins, but in the event of a singularity where limits are removed past the point where we know what to make of it.. it’s still tough to say what that means kinda by definition.
Friston has some insights which seem relevant to me though. He talks about his “theory of every thing” (humorously distinct from “theory of everything”), and explains that every thing that exists must necessarily resist entropic forces towards disintegration or they would cease to exist. So in that sense, everything that exists seeks to maintain its own existence -- including drops of oil in water.
Humans obviously tend to actively maintain their own existence, but also lineages of humans on longer timescales. It’s not clear to me what happens when these things conflict and what “lineage” actually means once transhuman stuff becomes possible.
When we think we want to go to the Chinese food restaurant, it’s because we want to eat the food. When we learn that it’s closed and don’t wanna go anymore, that’s not “value drift” to hold off, it’s learning that what we thought we wanted isn’t serving what we really want.
When we think we want to eat the Chinese food, it’s not because of some terminal “It’s just really good”. Get food poisoning once, and you’ll notice that there is indeed accounting for taste. “Yummy” becomes “Yucky” when we realize that the underlying value isn’t being delivered. I’ve even seen things as “innate and fundamental” as sexual orientation shift as it becomes clear that there is/isn’t value to be delivered.
How can you prevent the drift of value, but by sticking one’s head in the sand and refusing to learn that it’s not giving us what we want? How can we want to prevent this “value drift”, once we “know more and think faster” enough to understand the mechanism? What you describe as “getting the self-control first” is making sure to not get the knowledge.
“Realizing things” is not separate from “self-modification”, such that they can come in different orders. Realizing things is what modifies the self. Realizing that the Chinese restaurant is closed modifies your self. Realizing that your spouse has been cheating on you modifies your self. Realizing that Jews aren’t actually what you need them to be changes your self.
The Hitler who updates on what his Jew hatred brought him becomes a different Hitler, regardless of the order, so long as he does the update. Some orders may make it easier to accept the update than others, and therefore might make it more or less likely that he updates, but if he learns that the value is not delivered then he learns the value is not delivered. And if it’s true, and he doesn’t learn, then he has not CE’ed his V.
I agree that the process of learning always changes you. Tasting a food you have never tried before switches you from neutral-ish[1] to either liking the food or disliking the food; either one is a change.
This process is not linear. For example, there is food (coffee, blue cheese, beer, dry wine...) that tastes terrible on first try to most people, but if you stick with it, sometimes you reprogram your brain to start enjoying it. If you are already reprogrammed, the choice is already made, but if you are a beginner… does it make more sense to respect your current preferences and avoid the food, or to keep eating it anyway because you know this decision will make you happy after you reprogram your brain?
I guess in real life, many people simply get reprogrammed under social pressure. Stage one, they would prefer not to do it, but they have a stronger preference to be perceived as cool by their environment. Stage two, now they are reprogrammed and there is no conflict.
But suppose that you are perfectly above all social pressure, and you happen to not like beer at the moment. Is there a good reason for you to choose the reprogramming anyway? Sure, if you do, you will be happy that you did. But if you won’t, you won’t miss it. (Maybe curiosity is the remaining argument for the change? But without the social pressure, there is no reason to be curious about this specific thing.) So it seems to me that there are two different reflectively stable outcomes, and it depends on history where you end up.
We could even go further; with superior self-modification skills I could give myself a completely arbitrary preference, for example a sexual fetish for triangles. It seems silly now, but after modifying myself, and decorating my palace with paintings of triangles, I would probably be happy that I did it. I may even feel a little curious right now about what such absurd situation would be like. So should I modify myself that way? Seems like the answer is no, but maybe if I had too much time and got bored… well, I also don’t have a strong reason against doing that, except for some general “let’s not do arbitrary self-modifications, even if individually mostly harmless, the cumulative effect might dilute too much the things I care about now”.
Perhaps good and evil[2] are also acquired tastes; I am not sure about this part, but it feels plausible.
All new knowledge is self-modification, but not all self-modification is new (external) knowledge. You can also self-modify by resolving internal tensions. Or by changing a random connection in your brain. Also, if you obtain new knowledge in different order, the later information gets interpreted in the light of the former.
The all-knowing Hitler would know that his original reasons for hating Jews are no longer valid, but he might retain an aesthetic preference for doing so, for example because the very emotion of feeling superior to someone feels enjoyable.
Even people neutral about a specific food may have a meta preference about experimenting with new foods in general.
I am not going to provide an exact definition, but I think there is some generic desire for the world to be a nice and happy place also for others, versus just not caring about that and therefore effectively sacrificing the happiness of others for whatever things I do care about.
Thanks for the detailed response
In practice, this path dependency thing is indeed important, but it has a lot to do with our tendency to get lost and fail to find coherence.
For example, instead of beer what about heroin? Needles are icky, but boy will that change if you give it a shot! What happens in the longer term isn’t so simple as “more happy” though, and especially when the effects come from exogenous chemicals, we can’t really trust our initial pleasure to cache out in anything real.
This can make heroin very risky because people will often fail to learn that heroin injections are yucky again, but that is where the road coheres to. I don’t have any experience with heroin, but I have tried legally prescribed opioids a couple times and went through this arc. After the first time I couldn’t stop thinking about it for a month because it felt so good. Eventually though, my brain kinda recognized that this is not actually a good thing, I don’t actually want it, and when I eventually tried it again “just because” it wasn’t even enjoyable.
Beer and coffee are a lot more subtle and have context dependent social stuff going on, but “If you do you’ll be glad you did” is far from a sure thing.
I mean… I kinda vote “yes”. Because curiosity is important, the things you learn from experience are important, and this is a relatively harmless example to experiment with.
In my experience though, arbitrary modifications like this aren’t very stable. If there isn’t any actual value being delivered, and you don’t try to set up a labyrinth of motivations to not-look, people tend to learn that triangles aren’t so exciting as they had tricked themselves into believing.
A railroad spike to the brain will indeed indeed modify you, and is not well described as “learning new things!”. But resolving internal tensions usually is. And the railroad spike to the brain/random connection change is generally well described as losing things you have learned, or learning (expected) falsehoods.
This is another “probably in practice, but only because we don’t reach coherence” things. Maybe you consider the latter in terms of the former without going back to reinterpret the former in terms of the latter, but Bayes doesn’t justify this failure to propagate updates with any sort of path dependence.
My point is that what we think of as inscrutable aesthetic preferences are built upon implicit beliefs about the world, and updating the underlying structure changes the aesthetics. Hitler may like to feel superior, but his potential superiority is itself a fact about reality that he could update on.
What happens when you sit with the question “Are you superior?”?
There’s often an impulse to flinch away, and refuse to update, but when you do, things change.
Then how does one tell apant the true terminal values and instrumental ones? Does it mean that the CEV of an individual human is likely to be some combo of satisfaction of primitive values, fun-theoretic ones, idiosyncratic ones and of a way to instill decision-theoretic results (like coordinating with others in prisoner-like dilemmas) into our primitive brains? And how would the latter two value types be changeable? How would they change in AIs?
I don’t know.
So like, one time I was playing football with my cousins on Thanksgiving, and hurt my foot bad enough that I thought it was broken. As I dug into why reality was different than I wanted, at first there was always something underneath.
“I wish my foot wasn’t hurting. Of course I wish my foot wasn’t hurting, who wouldn’t? I acknowledge the fact that my foot is hurting. Why is it hurting? Because it’s broken, lol.”
So then “I wish my foot wasn’t broken”, why’s it broken? Oh, because I was playing football and tripped. Why’d I trip? Shit just happens, man. So, “Of course shit happens, so of course I broke my foot when I was playing football with my family, so of course it hurts.” What’s the problem? Nothing, actually. Suffering resolved.
In that case, my desire to not be in pain bottomed out at not wanting to have hurt myself unnecessarily, and “shit just happens”, but like… the fact that it was serving as a terminal in this context is still dependent on the fact that I didn’t believe I could do anything about it. If there had been some nootropic that cuts the rate of mistakes to 10%, then “Why did shit happen?” now has an answer of “Because you didn’t improve your brain function, dummy”, and we’re back to the races.
I know how to get to the practical bottom of things, for a given context. But as some sort of general case where we remove “all” practical limits… I dunno man. I’m not sure the question is well formed. I’m not sure it’s not.
I don’t know where it grounds out, just where it doesn’t. Which is useful on the margin, and maybe even large margins, but in the event of a singularity where limits are removed past the point where we know what to make of it.. it’s still tough to say what that means kinda by definition.
Friston has some insights which seem relevant to me though. He talks about his “theory of every thing” (humorously distinct from “theory of everything”), and explains that every thing that exists must necessarily resist entropic forces towards disintegration or they would cease to exist. So in that sense, everything that exists seeks to maintain its own existence -- including drops of oil in water.
Humans obviously tend to actively maintain their own existence, but also lineages of humans on longer timescales. It’s not clear to me what happens when these things conflict and what “lineage” actually means once transhuman stuff becomes possible.