I don’t have time to read the paper or even skim it really, just page through it. But I will, perhaps unwisely, voice my intuitive assessment, and then maybe people who actually read it, can correct me.
I find their concept to be sinister and dangerous. What are the actual consequences of “unbundling the personhood bundle”? It means, on the one hand, that you get to create entities that resemble people but which you don’t need to treat as people (good if you want intelligent slaves); on the other hand, you also get to create entities that aren’t really people at all, but which laws, customs and institutions will treat as people (good if you want to hasten the real “great replacement”).
A major reason why I respond negatively, is the line in the abstract about how this pragmatic attitude allows one to “creat[e] bespoke solutions for different contexts”. That’s corporate-speak, and I do not trust people who work for a mega-corporation and say they want to create customized concepts of personhood, whether they are lawyers or computer scientists.
Another reason is their pragmatist, relativist attitude to personhood. One of my persistent worries is that superintelligence will have the right values but the wrong ontology of personhood, and here these authors shrug their shoulders and say, meh, there aren’t real facts about that to discover anyway, just ever-shifting social conventions. If I had the time to do my due diligence on this paper, I would want to investigate the authors (I don’t know any of them) and find out where they are coming from, philosophically and professionally, so I could really identify the spirit in which the paper is written.
That’s what I derive from a superficial glance at the paper. I wish I had time to analyze and reflect on it properly, so that I could get the nuances right, and also have a more measured and less emotional response. But time is short, yet the issues are important, so, that’s my hasty response.
(I actually appreciate the emotion in the response, so thanks for including it)
One of my persistent worries is that superintelligence will have the right values but the wrong ontology of personhood
I would’ve expected the opposite phrasing (right ontology wrong values, cf. “the AI knows but doesn’t care”) so this caught my eye. Have you or anyone else written anything about this elsewhere you can point me to? I initially thought of Jan Kulveit’s essays (e.g. this or this) but upon re-skimming they don’t really connect to what you said.
“Tiling the solar system with smiley faces” used to be a canonical example of misalignment, and it could emerge from a combination of right values and very crudely wrong ontology, e.g. if the ontology can’t distinguish between actual happiness and pictures of happiness.
A more subtle example might be, what if humans are conscious and uploads aren’t. If an upload is as empty of genuine intentionality as a smiley face, you might have a causal model of conscious mind which is structurally correct in every particular, but which also needs to be implemented in the right kind of substrate to actually be conscious. If your ontology was missing that last detail, your aligned superintelligence might be profoundly correct in its theory of values, but could still lead to de-facto human extinction by being the Pied Piper of a mass migration of humanity into virtual spaces where all those hedons are only being simulated rather than being instantiated.
Interesting example. Tangentially I’m guessing believing in substrate dependence is part of some folks’ visceral dislike of Richard Ngo’s story The Gentle Romance, which was meant to be utopian. I mostly lean against substrate dependence and so don’t find your example persuasive, although Scott Aaronson’s monstrous edge cases do give me pause:
what if each person on earth simulated one neuron of your brain, by passing pieces of paper around. It took them several years just to simulate a single second of your thought processes. Would that bring your subjectivity into being? Would you accept it as a replacement for your current body?
If so, then what if your brain were simulated, not neuron-by-neuron, but by a gigantic lookup table? That is, what if there were a huge database, much larger than the observable universe (but let’s not worry about that), that hardwired what your brain’s response was to every sequence of stimuli that your sense-organs could possibly receive. Would that bring about your consciousness?
Let’s keep pushing: if it would, would it make a difference if anyone actually consulted the lookup table? Why can’t it bring about your consciousness just by sitting there doing nothing?
To these standard thought experiments, we can add more. Let’s suppose that, purely for error-correction purposes, the computer that’s simulating your brain runs the code three times, and takes the majority vote of the outcomes. Would that bring three “copies” of your consciousness into being? Does it make a difference if the three copies are widely separated in space or time—say, on different planets, or in different centuries? Is it possible that the massive redundancy taking place in your brain right now is bringing multiple copies of you into being?
Maybe my favorite thought experiment along these lines was invented by my former student Andy Drucker. In the past five years, there’s been a revolution in theoretical cryptography, around something called Fully Homomorphic Encryption (FHE), which was first discovered by Craig Gentry. What FHE lets you do is to perform arbitrary computations on encrypted data, without ever decrypting the data at any point. So, to someone with the decryption key, you could be proving theorems, simulating planetary motions, etc. But to someone without the key, it looks for all the world like you’re just shuffling random strings and producing other random strings as output.
You can probably see where this is going. What if we homomorphically encrypted a simulation of your brain? And what if we hid the only copy of the decryption key, let’s say in another galaxy? Would this computation—which looks to anyone in our galaxy like a reshuffling of gobbledygook—be silently producing your consciousness?
Obviously you’re not obliged to, but if you ever get round to looking into the GDM paper more deeply like you mentioned I’d be interested in what you have to say, as you might change my opinion on it.
I don’t have time to read the paper or even skim it really, just page through it. But I will, perhaps unwisely, voice my intuitive assessment, and then maybe people who actually read it, can correct me.
I find their concept to be sinister and dangerous. What are the actual consequences of “unbundling the personhood bundle”? It means, on the one hand, that you get to create entities that resemble people but which you don’t need to treat as people (good if you want intelligent slaves); on the other hand, you also get to create entities that aren’t really people at all, but which laws, customs and institutions will treat as people (good if you want to hasten the real “great replacement”).
A major reason why I respond negatively, is the line in the abstract about how this pragmatic attitude allows one to “creat[e] bespoke solutions for different contexts”. That’s corporate-speak, and I do not trust people who work for a mega-corporation and say they want to create customized concepts of personhood, whether they are lawyers or computer scientists.
Another reason is their pragmatist, relativist attitude to personhood. One of my persistent worries is that superintelligence will have the right values but the wrong ontology of personhood, and here these authors shrug their shoulders and say, meh, there aren’t real facts about that to discover anyway, just ever-shifting social conventions. If I had the time to do my due diligence on this paper, I would want to investigate the authors (I don’t know any of them) and find out where they are coming from, philosophically and professionally, so I could really identify the spirit in which the paper is written.
That’s what I derive from a superficial glance at the paper. I wish I had time to analyze and reflect on it properly, so that I could get the nuances right, and also have a more measured and less emotional response. But time is short, yet the issues are important, so, that’s my hasty response.
(I actually appreciate the emotion in the response, so thanks for including it)
I would’ve expected the opposite phrasing (right ontology wrong values, cf. “the AI knows but doesn’t care”) so this caught my eye. Have you or anyone else written anything about this elsewhere you can point me to? I initially thought of Jan Kulveit’s essays (e.g. this or this) but upon re-skimming they don’t really connect to what you said.
“Tiling the solar system with smiley faces” used to be a canonical example of misalignment, and it could emerge from a combination of right values and very crudely wrong ontology, e.g. if the ontology can’t distinguish between actual happiness and pictures of happiness.
A more subtle example might be, what if humans are conscious and uploads aren’t. If an upload is as empty of genuine intentionality as a smiley face, you might have a causal model of conscious mind which is structurally correct in every particular, but which also needs to be implemented in the right kind of substrate to actually be conscious. If your ontology was missing that last detail, your aligned superintelligence might be profoundly correct in its theory of values, but could still lead to de-facto human extinction by being the Pied Piper of a mass migration of humanity into virtual spaces where all those hedons are only being simulated rather than being instantiated.
Interesting example. Tangentially I’m guessing believing in substrate dependence is part of some folks’ visceral dislike of Richard Ngo’s story The Gentle Romance, which was meant to be utopian. I mostly lean against substrate dependence and so don’t find your example persuasive, although Scott Aaronson’s monstrous edge cases do give me pause:
Obviously you’re not obliged to, but if you ever get round to looking into the GDM paper more deeply like you mentioned I’d be interested in what you have to say, as you might change my opinion on it.