Although selfishness w.r.t. copies is a totally okay preference structure, rational agents (with a world-model like we have, and no preferences explicitly favoring conflict between copies) will want to precommit or self-modify so that their causal descendants will cooperate non-selfishly.
In fact, if there is a period where the copies don’t have distinguishing indexical information that greatly uncorrelates their decision algorithm, copies will even do the precommitting themselves.
Therefore, upon waking up and learning that I am a copy, but before learning much more, I will attempt to sign a contract with a bystander stating that if I do not act altruistically towards my other copies who have signed similar contracts, I have to pay them my life savings.
If signing a contract was all that we needed to coordinate well, we would already be coordinating as much as is useful now. We already have good strong reasons to want to coordinate for mutual benefit.
This is a good point—one could already sign a future-altruism-contract with someone, and it would already be an expected gain if it worked. But we only see approximations to this, like insurance or marriage. So unless my copies are enough more trustworthy and thoughtful of me to make this work, maybe something on the more efficacious self-modification end of the spectrum is actually necessary.
We don’t have the identity and value overlap with others that we’d have with copies. The contract would just be formalizing it. I think it’s a silly way of formalizing it. I respect my copies’ right to drift differently than I do, and will then cease cooperating as absolutely. I certainly don’t want to lose all of my assets in this case!
Moreover, when copying becomes the primary means of reproduction, caring for one’s copies becomes the ultimate in kin-selection. That puts a lot of evolutionary pressure on to favoring copy-cooperation. Imagine how siblings would care for each other if identical twins (triplets, N-tuples) were the norm.
Philosophically, I would want to value each of my copies equally, and I suspect that initially, my copies would be pretty altruistic towards each other. Using some mechanism to keep it that way, as Manfred suggests, seems appealing to me, but it isn’t clear how feasible it would be. I would expect that absent some such mechanism, I would gradually become less altruistic towards copies for psychological reasons: If I benefited another copy of myself at my own expense, I remember the expense and not the benefit, so even though I would endorse it as good for me in aggregate (if the benefit outweighed the expense), I would be trained not to do that via reinforcement learning. I expect that I would remain able to cooperate with copies pretty well for quite a long time, in the sense of coordinating for mutual benefit, since I would trust myself and there would therefore be lower enforcement costs to contracts, but I might fairly quickly stop being any more altruistic towards copies, in the sense of willingness to help them at my expense without an expectation of return, than I am towards close friends.
I would like to think that I would cooperate reasonably with my copies, especially when there is a strong reason to prioritize global values over selfish values.
However, in practice I would also expect that System 1 would still see copies as separate but related individuals rather than as myself, and this would limit the amount of cooperation that occurs. I might have to engage in some self-deceptive reasoning to accomplish selfishness, but the human brain is good at that (“I’ve been working harder than my copies—I deserve a little extra!”)
He isn’t me. He is a separate person that just happens to share all of the same memories and motivations that I have. I want to say that I wouldn’t even give this copy of me the time of day, but that would be rhetorical. In some ventures he would be my greatest friend, in others my worst enemy. (Interestingly I could accuratly tell which right now by application of decision theory to the variants of the prisoner’s delima.) But even when I choose to interfere in his affairs, it is not for directly self-serving reasons—I help him for the same reason I’d help a really close friend, I hurt him for the same reason I’d hinder a competitor.
My first thought (in response to the second question) is ‘immediately terminate myself, leaving the copy as the only valid continuation of my identity’.
Of course, it is questionable whether I would have the willpower to go through with it. I believe that my copy’s mind would constitute just as ‘real’ a continuation of my consciousness as would my own mind following a procedure that removed the memories of the past few days (or however long since the split) whilst leaving all else intact (which is of course just a contrived-for-the-sake-of-the-thought-experiment variety of the sort of forgetting that we undergo all the time), but I have trouble alieving it.
Kind of. I wouldn’t defect against my copy without his consent, but I would want the pool trimmed down to only a single version of myself (ideally whichever one had the highest expected future utility, all else equal). The copy, being a copy, should want the same thing. The only time I wouldn’t be opposed to the existence of multiple instances of myself would be if those instances could regularly synchronize their memories and experiences (and thus constitute more a single distributed entity with mere synchronization delays than multiple diverging entities).
Really? Can you say a little more about why you think you have that value? I guess I’m not convinced that it’s really a terminal value if it varies so widely across people of otherwise similar beliefs. Presumably that’s what lalartu meant as well, but I just don’t get it. I like myself, so I’d like more of myself in the world!
I think a big part of it is that I don’t really care about other people except instrumentally. I care terminally about myself, but only because I experience my own thoughts and feelings first-hand. If I knew I were going to be branched, then I’d care about both copies in advance as both are valid continuations of my current sensory stream. However, once the branch had taken place, both copies would immediately stop caring about the other (although I expect they would still practice altruistic behavior towards each other for decision-theoretic reasons). I suspect this has also influenced my sense of morality: I’ve never been attracted to total utilitarianism, as I’ve never been able to see why the existence of X people should be considered superior to the existence of Y < X equally satisfied people.
So yeah, that’s part of it, but not all of it (if that were the extent of it, I’d be indifferent to the existence of copies, not opposed to it). The rest is hard to put into words, and I suspect that even were I to succeed in doing so I’d only have succeeded in manufacturing a verbal rationalization. Part of it is instrumental, each copy would be a potential competitor, but that’s insufficient to explain my feelings on the matter. This wouldn’t be applicable to, say, the Many-Worlds Interpretation of quantum mechanics, and yet I’m still bothered by that interpretation as it implies constant branching of my identity. So in the end, I think that I can’t offer a verbal justification for this preference precisely because it’s a terminal preference.
If you had copies, how altruistic do you think you would be toward them?
Funny timing! Or, good Baader-Meinhoffing :P
Although selfishness w.r.t. copies is a totally okay preference structure, rational agents (with a world-model like we have, and no preferences explicitly favoring conflict between copies) will want to precommit or self-modify so that their causal descendants will cooperate non-selfishly.
In fact, if there is a period where the copies don’t have distinguishing indexical information that greatly uncorrelates their decision algorithm, copies will even do the precommitting themselves.
Therefore, upon waking up and learning that I am a copy, but before learning much more, I will attempt to sign a contract with a bystander stating that if I do not act altruistically towards my other copies who have signed similar contracts, I have to pay them my life savings.
If signing a contract was all that we needed to coordinate well, we would already be coordinating as much as is useful now. We already have good strong reasons to want to coordinate for mutual benefit.
This is a good point—one could already sign a future-altruism-contract with someone, and it would already be an expected gain if it worked. But we only see approximations to this, like insurance or marriage. So unless my copies are enough more trustworthy and thoughtful of me to make this work, maybe something on the more efficacious self-modification end of the spectrum is actually necessary.
We don’t have the identity and value overlap with others that we’d have with copies. The contract would just be formalizing it. I think it’s a silly way of formalizing it. I respect my copies’ right to drift differently than I do, and will then cease cooperating as absolutely. I certainly don’t want to lose all of my assets in this case!
Moreover, when copying becomes the primary means of reproduction, caring for one’s copies becomes the ultimate in kin-selection. That puts a lot of evolutionary pressure on to favoring copy-cooperation. Imagine how siblings would care for each other if identical twins (triplets, N-tuples) were the norm.
I think me and my copies would cooperate well and we would get a lot of utility from our cooperation.
Philosophically, I would want to value each of my copies equally, and I suspect that initially, my copies would be pretty altruistic towards each other. Using some mechanism to keep it that way, as Manfred suggests, seems appealing to me, but it isn’t clear how feasible it would be. I would expect that absent some such mechanism, I would gradually become less altruistic towards copies for psychological reasons: If I benefited another copy of myself at my own expense, I remember the expense and not the benefit, so even though I would endorse it as good for me in aggregate (if the benefit outweighed the expense), I would be trained not to do that via reinforcement learning. I expect that I would remain able to cooperate with copies pretty well for quite a long time, in the sense of coordinating for mutual benefit, since I would trust myself and there would therefore be lower enforcement costs to contracts, but I might fairly quickly stop being any more altruistic towards copies, in the sense of willingness to help them at my expense without an expectation of return, than I am towards close friends.
I would like to think that I would cooperate reasonably with my copies, especially when there is a strong reason to prioritize global values over selfish values.
However, in practice I would also expect that System 1 would still see copies as separate but related individuals rather than as myself, and this would limit the amount of cooperation that occurs. I might have to engage in some self-deceptive reasoning to accomplish selfishness, but the human brain is good at that (“I’ve been working harder than my copies—I deserve a little extra!”)
I’ve written about this previously on LessWrong:
I would want both copy and person who created it dead.
What would your copy want?
What if it was a near-copy without $fatalMedicalCondition?
My first thought (in response to the second question) is ‘immediately terminate myself, leaving the copy as the only valid continuation of my identity’.
Of course, it is questionable whether I would have the willpower to go through with it. I believe that my copy’s mind would constitute just as ‘real’ a continuation of my consciousness as would my own mind following a procedure that removed the memories of the past few days (or however long since the split) whilst leaving all else intact (which is of course just a contrived-for-the-sake-of-the-thought-experiment variety of the sort of forgetting that we undergo all the time), but I have trouble alieving it.
This is a lot more interesting a response if you would also agree with Lalartu in the more general case.
Kind of. I wouldn’t defect against my copy without his consent, but I would want the pool trimmed down to only a single version of myself (ideally whichever one had the highest expected future utility, all else equal). The copy, being a copy, should want the same thing. The only time I wouldn’t be opposed to the existence of multiple instances of myself would be if those instances could regularly synchronize their memories and experiences (and thus constitute more a single distributed entity with mere synchronization delays than multiple diverging entities).
Why would you want to actively avoid having a copy?
Because I terminally value the uniqueness of my identity.
Really? Can you say a little more about why you think you have that value? I guess I’m not convinced that it’s really a terminal value if it varies so widely across people of otherwise similar beliefs. Presumably that’s what lalartu meant as well, but I just don’t get it. I like myself, so I’d like more of myself in the world!
I think a big part of it is that I don’t really care about other people except instrumentally. I care terminally about myself, but only because I experience my own thoughts and feelings first-hand. If I knew I were going to be branched, then I’d care about both copies in advance as both are valid continuations of my current sensory stream. However, once the branch had taken place, both copies would immediately stop caring about the other (although I expect they would still practice altruistic behavior towards each other for decision-theoretic reasons). I suspect this has also influenced my sense of morality: I’ve never been attracted to total utilitarianism, as I’ve never been able to see why the existence of X people should be considered superior to the existence of Y < X equally satisfied people.
So yeah, that’s part of it, but not all of it (if that were the extent of it, I’d be indifferent to the existence of copies, not opposed to it). The rest is hard to put into words, and I suspect that even were I to succeed in doing so I’d only have succeeded in manufacturing a verbal rationalization. Part of it is instrumental, each copy would be a potential competitor, but that’s insufficient to explain my feelings on the matter. This wouldn’t be applicable to, say, the Many-Worlds Interpretation of quantum mechanics, and yet I’m still bothered by that interpretation as it implies constant branching of my identity. So in the end, I think that I can’t offer a verbal justification for this preference precisely because it’s a terminal preference.