This sounds a lot like people who strongly urge others to take on a life-changing decision (joining a cult of some kind, having children, whatever) by saying that once you go for it, you will never ever want to go back to the way things were before you took the plunge.
This may be true to whatever extent, and in the story that extent is absolute, but it doesn’t make for a very good sales pitch.
Can we get anything out of this analogy? If “once you join the cult, you’ll never want to go back to your pre-cult life” is unnapealing because there is something fundamentally wrong with cults, can we look for a similar bug in wireheading, perfect world simulations, and so on?
The pattern, “Once you do X you won’t want to not do X” isn’t inherently evil. Once you breathe oxygen you won’t want to not breathe oxygen.
I think the deeper problem has to do with identity. If doing X implies that I will suddenly stop caring about everything I am doing, have done, or will do… is it still me?
Breathing oxygen isn’t a choice, though. You have to go to great lengths (such as inserting yourself into an environment where it isn’t breathable, such as vacuum or deep water) to stop breathing it for more than a few minutes before your conscious control is overriden.
a better example might be trying a hobby and finding you like it
But that sounds nice! Noone wants the wireheading to be nice! Its supposed to be scary but they want it anyway so its even scarier. People wanting fun stuff isn’t scary its just nice and its not interesting.
The correct response is the one at the intersection of possible responses to both cases such that the thread devolves into massive meta fun and uncertain pseudo-sarcasm.
Well, a major bug in cults is that they take all your money and you spend the rest of your life working to further the cult’s interests. So perhaps the opportunity cost?
OTOH, it could be that something essential is missing—a cult is based on lies, an experience machine is full of zombies.
This sounds a lot like people who strongly urge others to take on a life-changing decision (joining a cult of some kind, having children, whatever) by saying that once you go for it, you will never ever want to go back to the way things were before you took the plunge.
This may be true to whatever extent, and in the story that extent is absolute, but it doesn’t make for a very good sales pitch.
Can we get anything out of this analogy? If “once you join the cult, you’ll never want to go back to your pre-cult life” is unnapealing because there is something fundamentally wrong with cults, can we look for a similar bug in wireheading, perfect world simulations, and so on?
The pattern, “Once you do X you won’t want to not do X” isn’t inherently evil. Once you breathe oxygen you won’t want to not breathe oxygen.
I think the deeper problem has to do with identity. If doing X implies that I will suddenly stop caring about everything I am doing, have done, or will do… is it still me?
The sunk cost fallacy may come into play as well.
“Once you stopped breathing oxygen you won’t want to breathe oxygen ever again.” is a more evil example.
Well, there is an adjustment period there.
Breathing oxygen isn’t a choice, though. You have to go to great lengths (such as inserting yourself into an environment where it isn’t breathable, such as vacuum or deep water) to stop breathing it for more than a few minutes before your conscious control is overriden.
You make a good argument that all those people who aren’t breathing are missing out :/
Seriously though, a better example might be trying a hobby and finding you like it so much you devote significant resources and time to it.
But that sounds nice! Noone wants the wireheading to be nice! Its supposed to be scary but they want it anyway so its even scarier. People wanting fun stuff isn’t scary its just nice and its not interesting.
Not sure if serious … ≖_≖
The correct response is the one at the intersection of possible responses to both cases such that the thread devolves into massive meta fun and uncertain pseudo-sarcasm.
Well, a major bug in cults is that they take all your money and you spend the rest of your life working to further the cult’s interests. So perhaps the opportunity cost?
OTOH, it could be that something essential is missing—a cult is based on lies, an experience machine is full of zombies.