The main disagreement between Aaronson’s idea and LW ideas seems to be this:
If any of these technologies—brain-uploading, teleportation, the Newcomb predictor, etc.—were actually realized, then all sorts of “woolly metaphysical questions” about personal identity and free will would start to have practical consequences. Should you fax yourself to Mars or not? Sitting in the hospital room, should you bet that the coin landed heads or tails? Should you expect to “wake up” as one of your backup copies, or as a simulation being run by the Newcomb Predictor? These questions all seem “empirical,” yet one can’t answer them without taking an implicit stance on questions that many people would prefer to regard as outside the scope of science.
(...)
As far as I can see, the only hope for avoiding these difficulties is if—because of chaos, the limits of quantum measurement, or whatever other obstruction—minds can’t be copied perfectly from one physical substrate to another, as can programs on standard digital computers. So that’s a possibility that this essay explores at some length. To clarify, we can’t use any philosophical difficulties that would arise if minds were copyable, as evidence for the empirical claim that they’re not copyable. The universe has never shown any particular tendency to cater to human philosophical prejudices! But I’d say the difficulties provide more than enough reason to care about the copyability question.
LW mostly prefers to bite the bullet on such questions, by using tools such as UDT. I’d be really curious to see Aaronson’s response to Wei’s UDT post.
As far as I can see, the only hope for avoiding these difficulties is if—because of chaos, the limits of quantum measurement, or whatever other obstruction—minds can’t be copied perfectly from one physical substrate to another, as can programs on standard digital computers.
Even if Aaronson’s speculation that human minds are not copyable turns out to be correct, that doesn’t rule out copyable minds being built in the future, either de novo AIs or what he (on page 58) calls “mockups” of human minds that are functionally close enough to the originals to fool their close friends. The philosophical problems with copyable minds will still be an issue for those minds, and therefore minds not being copyable can’t be the only hope of avoiding these difficulties.
To put this another way, suppose Aaronson definitively shows that according to quantum physics, minds of biological humans can’t be copied exactly. But how does he know that he is actually one of the original biological humans, and not for example a “mockup” living inside a digital simulation, and hence copyable? I think that is reason enough for him to directly attack the philosophical problems associated with copyable minds instead of trying to dodge them.
Wei, I completely agree that people should “directly attack the philosophical problems associated with copyable minds,” and am glad that you, Eliezer, and others have been trying to do that! I also agree that I can’t prove I’m not living in a simulation—nor that that fact won’t be revealed to me tomorrow by a being in the meta-world, who will also introduce me to dozens of copies of myself running in other simulations. But as long as we’re trading hypotheticals: what if minds (or rather, the sorts of minds we have) can only be associated with uncopyable physical substrates? What if the very empirical facts that we could copy a program, trace its execution, predict its outputs using an abacus, run the program backwards, in heavily-encrypted form, in one branch of a quantum computation, at one step per millennium, etc. etc., were to count as reductios that there’s probably nothing that it’s like to be that program—or at any rate, nothing comprehensible to beings such as us?
Again, I certainly don’t know that this is a reasonable way to think. I myself would probably have ridiculed it, before I realized that various things that confused me for years and that I discuss in the essay (Newcomb, Boltzmann brains, the “teleportation paradox,” Wigner’s friend, the measurement problem, Bostrom’s observer-counting problems...) all seemed to beckon me in that direction from different angles. So I decided that, given the immense perplexities associated with copyable minds (which you know as well as anyone), the possibility that uncopyability is essential to our subjective experience was at least worth trying to “steelman” (a term I learned here) to see how far I could get with it. So, that’s what I tried to do in the essay.
But as long as we’re trading hypotheticals: what if minds (or rather, the sorts of minds we have) can only be associated with uncopyable physical substrates?
If that turns out to be the case, I don’t think it would much diminish either my intellectual curiosity about how problems associated with mind copying ought to be solved nor the practical importance of solving such problems (to help prepare for a future where most minds will probably be copyable, even if my own isn’t).
various things that confused me for years and that I discuss in the essay (Newcomb, Boltzmann brains, the “teleportation paradox,” Wigner’s friend, the measurement problem, Bostrom’s observer-counting problems...) all seemed to beckon me in that direction from different angles
It seems likely that in the future we’ll be able to build minds that are very human-like, but copyable. For example we could take someone’s gene sequence, put them inside a virtual embryo inside a digital simulation, let it grow into an infant and then raise it in a virtual environment similar to a biological human child’s. I’m assuming that you don’t dispute this will be possible (at least in principle), but are saying that such a mind might not have the same kind of subjective experience as we do. Correct?
Now suppose we built such a mind using your genes, and gave it an upbringing and education similar to yours. Wouldn’t you then expect it to be puzzled by all the things that you mentioned above, except it would have to solves those puzzles in some way other than by saying “I can get around these confusions if I’m not copyable”? Doesn’t that suggest to you that there have to be solutions to those puzzles that do not involve “I’m not copyable” and therefore the existence of the puzzles shouldn’t have beckoned you in the direction of thinking that you’re uncopyable?
So I decided that, given the immense perplexities associated with copyable minds (which you know as well as anyone), the possibility that uncopyability is essential to our subjective experience was at least worth trying to “steelman” (a term I learned here) to see how far I could get with it.
If you (or somebody) eventually succeed in showing that uncopyability is essential to our subjective experience, that would mean that by introspecting on the quality of our subjective experience, we would be able to determine whether or not we are copyable, right? Suppose we take a copyable mind (such as the virtual Scott Aaronson clone mentioned above), make another copy of it, then turn one of the two copies into an uncopyable mind by introducing some freebits into it. Do you think these minds would be able to accurately report whether they are copyable, and if so, by what plausible mechanism?
(1) I agree that we can easily conceive of a world where most entities able to pass the Turing Test are copyable. I agree that it’s extremely interesting to think about what such a world would be like—and maybe even try to prepare for it if we can. And as for how the copyable entities will reason about their own existence—well, that might depend on the goals of whoever or whatever set them loose! As a simple example, the Stuxnet worm eventually deleted itself, if it decided it was on a computer that had nothing to do with Iranian centrifuges. We can imagine that each copy “knew” about the others, and “knew” that it might need to kill itself for the benefit of its doppelgangers. And as for why it behaved that way—well, we could answer that question in terms of the code, or in terms of the intentions of the people who wrote the code. Of course, if the code hadn’t been written by anyone, but was instead (say) the outcome of some evolutionary process, then we’d have to look for an explanation in terms of that process. But of course it would help to have the code to examine!
(2) You argue that, if I were copyable, then the copies would wonder about the same puzzles that the “uncopyable” version wonders about—and for that reason, it can’t be legitimate even to try to resolve those puzzles by assuming that I’m not copyable. Compare to the following argument: if I were a character in a novel, then that character would say exactly the same things I say for the same reasons, and wonder about the same things that I wonder about. Therefore, when reasoning about (say) physics or cosmology, it’s illegitimate even to make the tentative assumption that I’m not a character in a novel. This is a fun argument, but there are several possible responses, among them: haven’t we just begged the question, by assuming there is something it’s like to be a copyable em or a character in a novel? Again, I don’t declare with John Searle that there’s obviously nothing that it’s like, if you think there is then you need your head examined, etc. etc. On the other hand, even if I were a character in a novel, I’d still be happy to have that character assume it wasn’t a character—that its world was “real”—and see how far it could get with that assumption.
(3) No, I absolutely don’t think that we can learn whether we’re copyable or not by “introspecting on the quality of our subjective experience,” or that we’ll ever be able to do such a thing. The sort of thing that might eventually give us insight into whether we’re copyable or not would be understanding the effect of microscopic noise on the sodium-ion channels, whether the noise can be grounded in PMDs, etc. If you’ll let me quote from Sec. 2.1 of my essay: “precisely because one can’t decide between conflicting introspective reports, in this essay I’ll be exclusively interested in what can be learned from scientific observation and argument. Appeals to inner experience—including my own and the reader’s—will be out of bounds.”
And as for how the copyable entities will reason about their own existence
I’m not interested so much in how they will reason, but in how they should reason.
The sort of thing that might eventually give us insight into whether we’re copyable or not would be understanding the effect of microscopic noise on the sodium-ion channels, whether the noise can be grounded in PMDs, etc.
When you say “we” here, do you literally mean “we” or do you mean “biological humans”? Because I can see how understanding the effect of microscopic noise on the sodium-ion channels might give us insight into whether biological humans are copyable, but it doesn’t seem to tell us whether we are biological humans or for example digital simulations (and therefore whether your proposed solution to the philosophical puzzles is of any relevance to us). I thought you were proposing that if your theory is correct then we would eventually be able to determine that by introspection, since you said copyable minds might have no subjective experience or a different kind of subjective experience.
(1) Well, that’s the funny thing about “should”: if copyable entities have a definite goal (e.g., making as many additional copies as possible, taking over the world...), then we simply need to ask what form of reasoning will best help them achieve the goal. If, on the other hand, the question is, “how should a copy reason, so as to accord with its own subjective experience? e.g., all else equal, will it be twice as likely to ‘find itself’ in a possible world with twice as many copies?”—then we need some account of the subjective experience of copyable entities before we can even start to answer the question.
(2) Yes, certainly it’s possible that we’re all living in a digital simulation—in which case, maybe we’re uncopyable from within the simulation, but copyable by someone outside the simulation with “sysadmin access.” But in that case, what can I do, except try to reason based on the best theories we can formulate from within the simulation? It’s no different than with any “ordinary” scientific question.
(3) Yes, I raised the possibility that copyable minds might have no subjective experience or a different kind of subjective experience, but I certainly don’t think we can determine the truth of that possibility by introspection—or for that matter, even by “extrospection”! :-) The most we could do, maybe, is investigate whether the physical substrate of our minds makes them uncopyable, and therefore whether it’s even logically coherent to imagine a distinction between them and copyable minds.
The most we could do, maybe, is investigate whether the physical substrate of our minds makes them uncopyable, and therefore whether it’s even logically coherent to imagine a distinction between them and copyable minds.
If that’s the most you’re expecting to show at the end of your research program, then I don’t understand why you see it as a “hope” of avoiding the philosophical difficulties you mentioned. (I mean I have no problems with it as a scientific investigation in general, it’s just that it doesn’t seem to solve the problems that originally motivated you.) For example according to Nick Bostrom’s Simulation Argument, most human-like minds in our universe are digital simulations run by posthumans. How do you hope to conclude that the simulations “shouldn’t even be included in my reference class” if you don’t hope to conclude that you, personally, are not copyable?
What’s it like to be simulated in homomorphically encrypted form (http://en.wikipedia.org/wiki/Homomorphic_encryption)---so that someone who saw the entire computation (including its inputs and outputs), and only lacked a faraway decryption key, would have no clue that the whole thing is isomorphic to what your brain is doing?
What’s it like to be simulated by a reversible computer, and immediately “uncomputed”? Would you undergo the exact same set of experiences twice? Or once “forwards” and then once “backwards” (whatever that means)? Or, since the computation leaves no trace of its ever having happened, and is “just a convoluted implementation of the identity function,” would you not experience anything?
Once the code of your brain is stored in a computer, why would anyone even have to bother running the code to evoke your subjective experience? And what counts as running it? Is it enough to do a debugging trace with pen and paper?
Suppose that, purely for internal error-correction purposes, a computer actually “executes” you three times in parallel, then outputs the MAJORITY of the results. Is there now one conscious entity or three? (Or maybe 7, for every nonempty subset of executions?)
Crucially, unlike some philosophers (e.g. John Searle), I don’t pound the table and declare it “obvious” that there’s nothing that it’s like to be simulated in the strange ways above. All I say is that I don’t think I have any idea what it’s like, in even the same imperfect way that I can imagine what it’s like to be another human being (or even, say, an unclonable extraterrestrial) by analogy with my own case. And that’s why I’m not as troubled as some people are, if some otherwise-plausible cosmological theory predicts that the overwhelming majority of “copies” of me should be Boltzmann brains, computer simulations, etc. I view that as a sign, not that I’m almost certainly a copy (though I might be), but simply that I don’t yet know the right way to think about this issue, and maybe that there’s a good reason (lack of freebits??) why the supposed “copies” shouldn’t even be included in my reference class.
I really don’t like the term “LW consensus” (isn’t there a LW post about how you should separate out bundles of ideas and consider them separately because there’s no reason to expect the truth of one idea in a bundle to correlate strongly with the truth of the others? If there isn’t, there should be). I’ve been using “LW memeplex” instead to emphasize that these ideas have been bundled together for not necessarily systematically good reasons.
I think that last paragraph you quote needs the following extra bit of context:
To clarify, we can’t use any philosophical difficulties that would arise if minds were copyable, as evidence for the empirical claim that they’re not copyable. The universe has never shown any particular tendency to cater to human philosophical prejudices! But I’d say the difficulties provide more than enough reason to care about the copyability question.
… because otherwise it looks as if Aaronson is saying something really silly, which he isn’t.
If we could fax ourselves to Mars, or undergo uploading, then still wonder whether we’re still “us”—the same as we wonder now when such capabilities are just theoretical/hypotheticals—that should count as a strong indication that such questions are not very practically relevant, contrary to Aaronson’s assertion. Surely we’d need some legal rules, but the basis for those wouldn’t be much different than any basis we have now—we’d still be none the wiser about what identity means, even standing around with our clones.
For example, if we were to wonder about a question of “what effect will a foom-able AI have on our civilization”, surely asking after the fact would yield different answers to asking before. With copies / uploads etc., you and your perfect copy could hold a meeting contemplating who stays married with the wife, and still start from the same basis with the same difficulty of finding the “true” answer as if you’d discussed the topic with a pal roleplaying your clone, in the present time.
The main disagreement between Aaronson’s idea and LW ideas seems to be this:
LW mostly prefers to bite the bullet on such questions, by using tools such as UDT. I’d be really curious to see Aaronson’s response to Wei’s UDT post.
Even if Aaronson’s speculation that human minds are not copyable turns out to be correct, that doesn’t rule out copyable minds being built in the future, either de novo AIs or what he (on page 58) calls “mockups” of human minds that are functionally close enough to the originals to fool their close friends. The philosophical problems with copyable minds will still be an issue for those minds, and therefore minds not being copyable can’t be the only hope of avoiding these difficulties.
To put this another way, suppose Aaronson definitively shows that according to quantum physics, minds of biological humans can’t be copied exactly. But how does he know that he is actually one of the original biological humans, and not for example a “mockup” living inside a digital simulation, and hence copyable? I think that is reason enough for him to directly attack the philosophical problems associated with copyable minds instead of trying to dodge them.
Wei, I completely agree that people should “directly attack the philosophical problems associated with copyable minds,” and am glad that you, Eliezer, and others have been trying to do that! I also agree that I can’t prove I’m not living in a simulation—nor that that fact won’t be revealed to me tomorrow by a being in the meta-world, who will also introduce me to dozens of copies of myself running in other simulations. But as long as we’re trading hypotheticals: what if minds (or rather, the sorts of minds we have) can only be associated with uncopyable physical substrates? What if the very empirical facts that we could copy a program, trace its execution, predict its outputs using an abacus, run the program backwards, in heavily-encrypted form, in one branch of a quantum computation, at one step per millennium, etc. etc., were to count as reductios that there’s probably nothing that it’s like to be that program—or at any rate, nothing comprehensible to beings such as us?
Again, I certainly don’t know that this is a reasonable way to think. I myself would probably have ridiculed it, before I realized that various things that confused me for years and that I discuss in the essay (Newcomb, Boltzmann brains, the “teleportation paradox,” Wigner’s friend, the measurement problem, Bostrom’s observer-counting problems...) all seemed to beckon me in that direction from different angles. So I decided that, given the immense perplexities associated with copyable minds (which you know as well as anyone), the possibility that uncopyability is essential to our subjective experience was at least worth trying to “steelman” (a term I learned here) to see how far I could get with it. So, that’s what I tried to do in the essay.
If that turns out to be the case, I don’t think it would much diminish either my intellectual curiosity about how problems associated with mind copying ought to be solved nor the practical importance of solving such problems (to help prepare for a future where most minds will probably be copyable, even if my own isn’t).
It seems likely that in the future we’ll be able to build minds that are very human-like, but copyable. For example we could take someone’s gene sequence, put them inside a virtual embryo inside a digital simulation, let it grow into an infant and then raise it in a virtual environment similar to a biological human child’s. I’m assuming that you don’t dispute this will be possible (at least in principle), but are saying that such a mind might not have the same kind of subjective experience as we do. Correct?
Now suppose we built such a mind using your genes, and gave it an upbringing and education similar to yours. Wouldn’t you then expect it to be puzzled by all the things that you mentioned above, except it would have to solves those puzzles in some way other than by saying “I can get around these confusions if I’m not copyable”? Doesn’t that suggest to you that there have to be solutions to those puzzles that do not involve “I’m not copyable” and therefore the existence of the puzzles shouldn’t have beckoned you in the direction of thinking that you’re uncopyable?
If you (or somebody) eventually succeed in showing that uncopyability is essential to our subjective experience, that would mean that by introspecting on the quality of our subjective experience, we would be able to determine whether or not we are copyable, right? Suppose we take a copyable mind (such as the virtual Scott Aaronson clone mentioned above), make another copy of it, then turn one of the two copies into an uncopyable mind by introducing some freebits into it. Do you think these minds would be able to accurately report whether they are copyable, and if so, by what plausible mechanism?
(1) I agree that we can easily conceive of a world where most entities able to pass the Turing Test are copyable. I agree that it’s extremely interesting to think about what such a world would be like—and maybe even try to prepare for it if we can. And as for how the copyable entities will reason about their own existence—well, that might depend on the goals of whoever or whatever set them loose! As a simple example, the Stuxnet worm eventually deleted itself, if it decided it was on a computer that had nothing to do with Iranian centrifuges. We can imagine that each copy “knew” about the others, and “knew” that it might need to kill itself for the benefit of its doppelgangers. And as for why it behaved that way—well, we could answer that question in terms of the code, or in terms of the intentions of the people who wrote the code. Of course, if the code hadn’t been written by anyone, but was instead (say) the outcome of some evolutionary process, then we’d have to look for an explanation in terms of that process. But of course it would help to have the code to examine!
(2) You argue that, if I were copyable, then the copies would wonder about the same puzzles that the “uncopyable” version wonders about—and for that reason, it can’t be legitimate even to try to resolve those puzzles by assuming that I’m not copyable. Compare to the following argument: if I were a character in a novel, then that character would say exactly the same things I say for the same reasons, and wonder about the same things that I wonder about. Therefore, when reasoning about (say) physics or cosmology, it’s illegitimate even to make the tentative assumption that I’m not a character in a novel. This is a fun argument, but there are several possible responses, among them: haven’t we just begged the question, by assuming there is something it’s like to be a copyable em or a character in a novel? Again, I don’t declare with John Searle that there’s obviously nothing that it’s like, if you think there is then you need your head examined, etc. etc. On the other hand, even if I were a character in a novel, I’d still be happy to have that character assume it wasn’t a character—that its world was “real”—and see how far it could get with that assumption.
(3) No, I absolutely don’t think that we can learn whether we’re copyable or not by “introspecting on the quality of our subjective experience,” or that we’ll ever be able to do such a thing. The sort of thing that might eventually give us insight into whether we’re copyable or not would be understanding the effect of microscopic noise on the sodium-ion channels, whether the noise can be grounded in PMDs, etc. If you’ll let me quote from Sec. 2.1 of my essay: “precisely because one can’t decide between conflicting introspective reports, in this essay I’ll be exclusively interested in what can be learned from scientific observation and argument. Appeals to inner experience—including my own and the reader’s—will be out of bounds.”
I’m not interested so much in how they will reason, but in how they should reason.
When you say “we” here, do you literally mean “we” or do you mean “biological humans”? Because I can see how understanding the effect of microscopic noise on the sodium-ion channels might give us insight into whether biological humans are copyable, but it doesn’t seem to tell us whether we are biological humans or for example digital simulations (and therefore whether your proposed solution to the philosophical puzzles is of any relevance to us). I thought you were proposing that if your theory is correct then we would eventually be able to determine that by introspection, since you said copyable minds might have no subjective experience or a different kind of subjective experience.
(1) Well, that’s the funny thing about “should”: if copyable entities have a definite goal (e.g., making as many additional copies as possible, taking over the world...), then we simply need to ask what form of reasoning will best help them achieve the goal. If, on the other hand, the question is, “how should a copy reason, so as to accord with its own subjective experience? e.g., all else equal, will it be twice as likely to ‘find itself’ in a possible world with twice as many copies?”—then we need some account of the subjective experience of copyable entities before we can even start to answer the question.
(2) Yes, certainly it’s possible that we’re all living in a digital simulation—in which case, maybe we’re uncopyable from within the simulation, but copyable by someone outside the simulation with “sysadmin access.” But in that case, what can I do, except try to reason based on the best theories we can formulate from within the simulation? It’s no different than with any “ordinary” scientific question.
(3) Yes, I raised the possibility that copyable minds might have no subjective experience or a different kind of subjective experience, but I certainly don’t think we can determine the truth of that possibility by introspection—or for that matter, even by “extrospection”! :-) The most we could do, maybe, is investigate whether the physical substrate of our minds makes them uncopyable, and therefore whether it’s even logically coherent to imagine a distinction between them and copyable minds.
If that’s the most you’re expecting to show at the end of your research program, then I don’t understand why you see it as a “hope” of avoiding the philosophical difficulties you mentioned. (I mean I have no problems with it as a scientific investigation in general, it’s just that it doesn’t seem to solve the problems that originally motivated you.) For example according to Nick Bostrom’s Simulation Argument, most human-like minds in our universe are digital simulations run by posthumans. How do you hope to conclude that the simulations “shouldn’t even be included in my reference class” if you don’t hope to conclude that you, personally, are not copyable?
What would make them “count as reductios that there’s probably nothing that it’s like to be that program”, and how?
Alright, consider the following questions:
What’s it like to be simulated in homomorphically encrypted form (http://en.wikipedia.org/wiki/Homomorphic_encryption)---so that someone who saw the entire computation (including its inputs and outputs), and only lacked a faraway decryption key, would have no clue that the whole thing is isomorphic to what your brain is doing?
What’s it like to be simulated by a reversible computer, and immediately “uncomputed”? Would you undergo the exact same set of experiences twice? Or once “forwards” and then once “backwards” (whatever that means)? Or, since the computation leaves no trace of its ever having happened, and is “just a convoluted implementation of the identity function,” would you not experience anything?
Once the code of your brain is stored in a computer, why would anyone even have to bother running the code to evoke your subjective experience? And what counts as running it? Is it enough to do a debugging trace with pen and paper?
Suppose that, purely for internal error-correction purposes, a computer actually “executes” you three times in parallel, then outputs the MAJORITY of the results. Is there now one conscious entity or three? (Or maybe 7, for every nonempty subset of executions?)
Crucially, unlike some philosophers (e.g. John Searle), I don’t pound the table and declare it “obvious” that there’s nothing that it’s like to be simulated in the strange ways above. All I say is that I don’t think I have any idea what it’s like, in even the same imperfect way that I can imagine what it’s like to be another human being (or even, say, an unclonable extraterrestrial) by analogy with my own case. And that’s why I’m not as troubled as some people are, if some otherwise-plausible cosmological theory predicts that the overwhelming majority of “copies” of me should be Boltzmann brains, computer simulations, etc. I view that as a sign, not that I’m almost certainly a copy (though I might be), but simply that I don’t yet know the right way to think about this issue, and maybe that there’s a good reason (lack of freebits??) why the supposed “copies” shouldn’t even be included in my reference class.
I really don’t like the term “LW consensus” (isn’t there a LW post about how you should separate out bundles of ideas and consider them separately because there’s no reason to expect the truth of one idea in a bundle to correlate strongly with the truth of the others? If there isn’t, there should be). I’ve been using “LW memeplex” instead to emphasize that these ideas have been bundled together for not necessarily systematically good reasons.
OK, replaced with “LW ideas”.
I think that last paragraph you quote needs the following extra bit of context:
… because otherwise it looks as if Aaronson is saying something really silly, which he isn’t.
Good point, thanks! Added that bit.
If we could fax ourselves to Mars, or undergo uploading, then still wonder whether we’re still “us”—the same as we wonder now when such capabilities are just theoretical/hypotheticals—that should count as a strong indication that such questions are not very practically relevant, contrary to Aaronson’s assertion. Surely we’d need some legal rules, but the basis for those wouldn’t be much different than any basis we have now—we’d still be none the wiser about what identity means, even standing around with our clones.
For example, if we were to wonder about a question of “what effect will a foom-able AI have on our civilization”, surely asking after the fact would yield different answers to asking before. With copies / uploads etc., you and your perfect copy could hold a meeting contemplating who stays married with the wife, and still start from the same basis with the same difficulty of finding the “true” answer as if you’d discussed the topic with a pal roleplaying your clone, in the present time.