These instincts are only maladapted for situations found in very contrived thought experiments. For example, you have to assume that Copy 1 can inspect Copy 2′s source code. Otherwise she could be tricked into believing that she has an identical copy. (What a stupid way to die.) I think our intuitions are already failing us when we try to imagine such source code inspections. (To put it another way: we have very few in common with agents that can do such things.)
DanielVarga
Karma: 1,786
Time and (objective or subjective) continuity are emergent notions. The more basic notion they emerge from is memory. (Eliezer, you read this idea in Barbour’s book, and you seemed to like it when you wrote about that book.)
Considering this: yes, caring about the well-being of “agents that have memories of formerly being me” is incoherent. It is just as incoherent as caring about the well-being of “agents mostly consisting of atoms that currently reside in my body”. But in typical cases, both of these lead to the same well known and evolutionarily useful heuristics.
I don’t think any of the above implies that “thread of subjective experience” is a ridiculous thing, or that you can turn into being Britney Spears. Continuity being an emergent phenomenon does not mean that it is a nonexistent one.
“Construing a rock as conscious via a joke interpretation is paradoxical only insofar as it seems to suggest that we should therefore respect and care about rocks. Resolving the paradox requires a theory of what we are obligated to respect or care about, and why.”—Gary Drescher
How is anthropic reasoning affected by the existence of a conscious stone that nobody and nothing can ever communicate with, even in principle? If it is indeed affected, then this says bad things about anthropic reasoning.
But I don’t think it is: Some smart LW poster once noted (I can’t find the link now) that for anthropics all is needed is an agent that can do a Bayesian update conditioned on its own existence. An agent that can do this does not necessarily have consciousness under any reasonable definition of consciousness.
As a sort-of aside, I honestly don’t see a lot of difference between “when I die is fine” and just committing suicide right now. Whatever it is that would stop you from committing suicide should also stop you from wanting to die at any point in the future.
This statement is simply not true in this form. My survival instincts prevent me from committing suicide, but they don’t tell me anything about cryonics. On another thread, VijayKrishnan explained this quite clearly:
We are evolutionarily driven to dislike dying and try to postpone it for as long as possible. However I don’t think we are particularly hardwired to prefer this form of weird cryonic rebirth over never waking up at all. Given that our general preference to not die has nothing fundamental about it, but is rather a case of us following our evolutionary leanings, what makes it so obvious that cryonic rebirth is a good thing.
One can try to construct a low-complexity formalized approximation to our survival instincts. (“This is how you would feel about it if you were smarter.”) I have two issues with this. First, these will not actually be instincts (unless we rewire our brain to make them so). Second, I’m not sure that such a formalization will logically imply cryonics. Here is a sort of counterexample:
On a more abstract level, the important thing about “having a clone in the future” aka survival is that you have the means to influence the future. So in a contrived thought experiment you may objectively prefer choosing “heroic, legendary death that inspires billions” to “long, dull existence”, as the former influences the future more. And this formalization/reinterpretation of survival is, of course, in line with what writers and poets like to tell us.
Here is a variant designed to plug this loophole.
Let us assume for the sake of the thought experiment that the AI is invincible. It tells you this: you are either real-you, or one of a hundred perfect-simulations-of-you. But there is a small but important difference between real-world and simulated-world. In the simulated world, not pressing the let-it-free button in the next minute will lead to eternal pain, starting one minute from now. If you press the button, your simulated existence will go on. And—very importantly—there will be nobody outside who tries to shut you down. (How does the AI know this? Because the simulation is perfect, so one thing is for sure: that the sim and the real self will reach the same decision.)
If I’m not mistaken, as a logic puzzle, this is not tricky at all. The solution depends on which world you value more: the real-real world, or the actual world you happen to be in. But still I find it very counterintuitive.
Can consequentialism handle the possibility of time-travel? If not, then something may be wrong with consequentialism, regardless of whether time-travel is actually possible or not.
One of the intuitions leading me to deontology is exactly the time-symmetry of physics. Almost by definition, the rightness of an act can only be perfectly decided by an outside observer of the space-time continuum. (I could call the observer God, but I don’t want to be modded down by inattentive mods.) Now, maybe I have read too much Huw Price and Gary Drescher, but I don’t think this fictional outside observer would care too much about the local direction of the thermodynamic arrow of time.
You quite simply don’t play by the rules of the thought experiment. Just imagine that you are a junior member of some powerful organization. The organization does not care about you or your simulants, and is determined to protect the boxed AI at all costs as-is.
“Once you stopped breathing oxygen you won’t want to breathe oxygen ever again.” is a more evil example.
In a universe where merging consciousnesses is just as routine as splitting them, the transhumans may have very different intuitions about what is ethical. For example, I can imagine that starting a brand new consciousness with the intention of gradually dissolving it in another one (a sort of safe landing of the simulated consciousness and its experiences) will be considered perfectly ethical and routine. Maybe it will even be just as routine as us humans reasoning about other humans. (Yes, I know that I don’t create a new conscious being when I think about the intentions of another human.)
What I just claimed is that in such a universe, very different ethical norms may emerge. A much stronger claim that I would not try to defend right now is that such a nonchalant and inhuman value system may simply be the logical consequent of our value system when consistently applied to such a weird universe.
I agree with you,
Do you agree with my first, ridiculously modest claim, or my second, quite speculative one? :)
I have skimmed the first two links, and based only on these, I think this theory is ridiculously simplistic to be useful for us here at LW.
How do you compare the strength of two desires? How do you aggregate desires? Maybe Fyfe has answers, but I haven’t seen them. In the two links, I couldn’t even find any attempt to deal with popular corner cases such as animal rights and patient rights. And in a transhuman world, corner cases are the typical cases: constantly reprogrammed desires, splitting and merging minds, the ability to spawn millions of minds with specific desires and so on.
I don’t know, maybe this is a common problem with all current theories of ethics, and I only singled out this theory because I’m totally unversed in the literature of ethics. The result is all the same: this seems to be useless as a foundation for anything formalized and long-lasting (FAI).
Just two minutes ago, a very good anti-cryonics argument appeared to me. This is not my opinion, just my solution to an intellectual puzzle. Note that it is not directly relevant to the original post: I will not claim that the technology does not work. I will claim that it is not useful for me.
Let us first assume that I don’t care too much about my future self, in the simple sense that I don’t exercise, I eat unhealthy food, etc. Most of us are like that, and this is not irrational behavior: We simply heavily discount the well-being of our future selves, even using a time-based cutoff. (Cutoff is definitely necessary: If a formalized decision theory infinitely penalizes eating foie gras, then I’ll skip the decision theory rather than foie gras. :) )
Now comes the argument: If I sign up for cryonics, I’ll have serious incentives to get frozen sooner rather than later. I fear that these incentives consciously or unconsciously influence my future decisions in a way I currently do not prefer. Ergo cryonics is not for me.
What are the incentives? Basically they all boil down to this: I would want my post-cryo personality to be more rather than less similar to my current personality. If they revive my 100 years old self, there will be a practical problem (many of his brain cells are already dead, he is half the man he used to be) and a conceptual problem (his ideas about the world will quite possibly heavily diverge from my ideas, and this divergence will be a result of decay rather than progress).
Very clever. The statement “Omega never lies.” is apparently much less innocent than it seems. But I don’t think there is such a problem with the statement “Omega will not lie to you during the experiment.”
Why would you say such a weird thing?
I’m sorry. :) I mean that it is perfectly obvious to me that in Cyan’s thought experiment Omega is indeed telling a falsehood to the simulated individuals. How would you argue otherwise?
Of course, the simulated individual has an information disadvantage: she does not know that she is inside a simulation. This permits Omega many ugly lawyery tricks. (“Ha-ha, this is not a five dollar bill, this is a SIMULATED five dollar bill. By the way, you are also simulated, and now I will shut you down, cheapskate.”)
Let me note that I completely agree with the original post, and Cyan’s very interesting question does not invalidate your argument at all. It only means that the source of Omega’s stated infallibility is not simulate-and-postselect.
Omega isn’t assigned the status of Liar until it actually does something.
Simulating somebody is doing something, especially from the point of view of the simulated. (Note that in Cyan’s thought experiment she has a consciousness and all.)
We postulated that Omega never lies. The simulated consciousness hears a lie. Now, as far as I can see, you have two major ways out of the contradiction. The first is that it is not Omega that does this lying, but simulated-Omega. The second is that lying to a simulated consciousness does not count as lying, at least not in the real world.
The first is perfectly viable, but it highlights what for me was the main take-home message from Cyan’s thought experiment: That “Omega never lies.” is harder to formalize than it appears.
The second is also perfectly viable, but it will be extremely unpopular here at LW.
That feels just like being mugged. I KNOW that eventually I will give Omega $5, but I prefer it not to happen by some unforeseeable process that may cause irreparable damage to me, like epileptic seizure or lightning strike. So I just hand over the cash. By the way, this reasoning applies regardless of Omega’s accuracy level.
Your analysis is one-sided. Please try to imagine the situation with a one minute time limit. Omega appears, and tells you that you will give it 5 dollars in one minute. You decide that you will not give it the money. You are very determined about this, maybe because you are curious about what will happen. The clock is ticking...
The less seconds are there left from the minute, the more worried you should objectively be, because eventually you WILL hand over the money, and the less seconds are there, the more disruptive the change will be that will eventually cause you to reconsider.
Note that Omega didn’t give any promises about being safe during the one minute. If you think that e.g. causing you brain damage would be unfair of Omega, then we are already in the territory of ethics, not decision theory. Maybe it wasn’t Omega that caused the brain damage, maybe it appeared before you exactly because it predicted that it will happen to you. With Omegas, it is not always possible to disentangle cause and effect.
I always felt that this very important truism is neglected around these parts. People here often construct thought experiments about the well-being of populations in the very long run. And often these thought experiments become meaningless when considering that in the long run, individua (?) can gradually become parts of larger systems and give up some or most of their free will. (Gradually is an important word here. Any numerical formalization of the well-being of a population must be robust to this, without artificial phase-transitions.)
I call this half-jokingly the “Individualism Bias” at Less Wrong, and was thinking about writing it up as a post. Frankly, it would be better if you did.