Yes, but why should my current preferences be binding on my future selves? They presumably know more than I do. I would hate to be bound by the preferences of my 9-year-old self with regards to, say, cooties. Or, to put it differently: I have a preference in the matter, but I’m not convinced it is strong enough to require binding self-modification.
I also have this problem with your scenario: Your “no matter how far” presupposes that I can put a limit on divergence: To wit, the copies cannot diverge far enough to work around my modification. This assumption may be unwarranted. It seems to amount to saying that I am able to decide “I will never defect against myself” and make it stick; but in this formulation it doesn’t look anywhere near so convincing as talking of ‘self-modification’. I don’t think speaking of self-modification is useful here; you should rather talk of making decisions, which is a process where we have actual experience.
That’s irrelevant, because their change in preference is not caused by additional knowledge, but due to a quirk in how humans make decisions. We never had mind copying in our EEA, so we make decisions mostly by valuing our own anticipated future experiences. In other words, we’re more like the first picture in http://lesswrong.com/lw/116/the_domain_of_your_utility_function/ (Presumably because that works well enough when mind copying isn’t possible, and is computationally cheaper, or just an easier solution for evolution to find.)
I don’t think speaking of self-modification is useful here; you should rather talk of making decisions, which is a process where we have actual experience.
I don’t understand this. As long as we’re talking about mind copying, why shouldn’t I talk about self-modification?
ETA: Perhaps you mean that I should consider the inconvenient possible world where mind copying is possible, but self-modification isn’t? In that case, yes, you may not be able to make “I will never defect against myself” stick.
But even in that world, you will be in competition against other minds, some of whom may be able to make it stick, and they will have a competitive advantage against you, since their copies will be able to better cooperate with each other. I don’t know if that argument moves you at all.
No, what I mean is that you should taboo “self-modification” and see what happens to your argument. If I decide, today, that I will go on a diet, is that not a weaker form of self-modifying? It is an attempt to bind future selves to a decision made today. Granted, it is a weak binding, as we all know; but to say “self-modification” is just to dismiss that difficulty, assuming that in the future we can overcome akrasia. Well, the future will contain many wonderful things, but I’m not convinced a cure for weak-willedness is among them! So “self-modification” becomes, when tabooed, “decide, with additional future inventions making me able to overcome the natural temptation to re-decide”; and I think this is a much more useful formulation. The reason is that we all have some experience with deciding to do something, and can perhaps form some impression of how much help we’re going to need from the future inventions; while we have zero experience with this magical “self-modification”, and can form no intuition of how powerful it is.
We do have some experience with self-modification, in the form of self-modifying programs. That is, programs that either write directly into their own executable memory, or (since modern CPUs tend to prohibit this) write out an executable file and then call exec on it.
Yes, but why should my current preferences be binding on my future selves? They presumably know more than I do. I would hate to be bound by the preferences of my 9-year-old self with regards to, say, cooties. Or, to put it differently: I have a preference in the matter, but I’m not convinced it is strong enough to require binding self-modification.
I also have this problem with your scenario: Your “no matter how far” presupposes that I can put a limit on divergence: To wit, the copies cannot diverge far enough to work around my modification. This assumption may be unwarranted. It seems to amount to saying that I am able to decide “I will never defect against myself” and make it stick; but in this formulation it doesn’t look anywhere near so convincing as talking of ‘self-modification’. I don’t think speaking of self-modification is useful here; you should rather talk of making decisions, which is a process where we have actual experience.
That’s irrelevant, because their change in preference is not caused by additional knowledge, but due to a quirk in how humans make decisions. We never had mind copying in our EEA, so we make decisions mostly by valuing our own anticipated future experiences. In other words, we’re more like the first picture in http://lesswrong.com/lw/116/the_domain_of_your_utility_function/ (Presumably because that works well enough when mind copying isn’t possible, and is computationally cheaper, or just an easier solution for evolution to find.)
I don’t understand this. As long as we’re talking about mind copying, why shouldn’t I talk about self-modification?
ETA: Perhaps you mean that I should consider the inconvenient possible world where mind copying is possible, but self-modification isn’t? In that case, yes, you may not be able to make “I will never defect against myself” stick.
But even in that world, you will be in competition against other minds, some of whom may be able to make it stick, and they will have a competitive advantage against you, since their copies will be able to better cooperate with each other. I don’t know if that argument moves you at all.
No, what I mean is that you should taboo “self-modification” and see what happens to your argument. If I decide, today, that I will go on a diet, is that not a weaker form of self-modifying? It is an attempt to bind future selves to a decision made today. Granted, it is a weak binding, as we all know; but to say “self-modification” is just to dismiss that difficulty, assuming that in the future we can overcome akrasia. Well, the future will contain many wonderful things, but I’m not convinced a cure for weak-willedness is among them! So “self-modification” becomes, when tabooed, “decide, with additional future inventions making me able to overcome the natural temptation to re-decide”; and I think this is a much more useful formulation. The reason is that we all have some experience with deciding to do something, and can perhaps form some impression of how much help we’re going to need from the future inventions; while we have zero experience with this magical “self-modification”, and can form no intuition of how powerful it is.
We do have some experience with self-modification, in the form of self-modifying programs. That is, programs that either write directly into their own executable memory, or (since modern CPUs tend to prohibit this) write out an executable file and then call exec on it.
But anyway, I think I get your point.