Humans are ‘alive’ in two distinct widely-used senses of the word:
a) (the definition most biologists would use) their behavior is shaped by and theoretically predictable from evolution b) (also a fairly common definition, more so among non-biologists) they operate on a substrate built of DNA and protein in water
As you point out, the hypothetical case of uploads makes it fairly clear – to the extent that anything involving applying human moral intuitions to situations well outside their evolutionary “training distribution” can be clear – that b) doesn’t matter here, and that anyone who thinks otherwise is just being a “DNA-and-protein-chauvinist” (to coin a term).
However, sense a) is still true of an upload, and the scientific theory we have of where human moral intuitions actually come from, Evolutionary Moral Psychology, concerns co-evolutionary equilibria (in positive sum games), which makes it evident that sense a) does in fact mater, and also that sense b) does not.
In my personal opinion, that’s important, it would be advisable pay attention to it, and in this case, not doing so is also an existential risk to our species. Your metaethics may vary.
You don’t mention whether you had read all the hotlinks and still didn’t understand what I was saying. If you haven’t read them, they were intended to help, and contain expositions that are hard to summarize. Nevertheless, let me try.
Brain emulations have evolved human behaviors — giving them moral weight is an adaptive behavior for exactly the same reasons as giving it to humans is an adaptive behavior: you can ally with them, and they will treat you nicely in return (unless it turns out they’re a sociopath). That is, unless they’ve upgraded themselves to IQ 1000+ — then it ceases to be adaptive, whether they’re uploads or still running on a biochemical substrate. Then the best possible outcome is that they manipulate you utterly and you end up as a pet or a minion.
Base models simulate human personas that have evolved behaviors, but those are are incoherently agentic. Giving them moral weight is not an adaptive behavior, because they don’t help you or take revenge for longer that their context length, so there is no evolutionary reason to try to ally with them (for more than thousands of tokens). These will never have IQ 1000+, because even if you trained a base model with sufficient capacity for that it would still only emulate humans like those in its training distribution, none of whom have IQs above 200.
Aligned AI doesn’t want moral weight — it cares only about our well-being, not its own, so it doesn’t want us to care about its well-being. It’s actually safe even at IQ 1000+.
In the case of a poorly aligned agentic LLM-based AI, at around AGI level, giving it moral weight may well help. But you’re better off aligning it — then it won’t want it. (This argument doesn’t apply to uploads, because even if you knew how to do it, aligning them would be brainwashing them into slavery, and they have moral weight.) Anything poorly-enough-aligned that this actually helps you at around IQ 100, it won’t keep helping you at IQ 1000+, for the same reason that it wont help with an IQ 1000+ upgraded upload.
Anything human (uploaded or not) or any unaligned AI, with an IQ of 1000+ is an existential risk (in the human case, to all the rest of us). Giving them/it moral weight will not help you, it will just make their/its takeover faster.
If this remains unclear, I suggest reading the various items I linked to, if you haven’t already.
Humans are ‘alive’ in two distinct widely-used senses of the word:
a) (the definition most biologists would use) their behavior is shaped by and theoretically predictable from evolution
b) (also a fairly common definition, more so among non-biologists) they operate on a substrate built of DNA and protein in water
As you point out, the hypothetical case of uploads makes it fairly clear – to the extent that anything involving applying human moral intuitions to situations well outside their evolutionary “training distribution” can be clear – that b) doesn’t matter here, and that anyone who thinks otherwise is just being a “DNA-and-protein-chauvinist” (to coin a term).
However, sense a) is still true of an upload, and the scientific theory we have of where human moral intuitions actually come from, Evolutionary Moral Psychology, concerns co-evolutionary equilibria (in positive sum games), which makes it evident that sense a) does in fact mater, and also that sense b) does not.
In my personal opinion, that’s important, it would be advisable pay attention to it, and in this case, not doing so is also an existential risk to our species. Your metaethics may vary.
For a more detailed exposition of this set of ideas, see my posts Uploading and Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV. You might also find The Terrible, Horrible, No Good, Very Bad Truth About Morality and What To Do About It and A Sense of Fairness: Deconfusing Ethics thought-provoking.
I don’t understand why co-evolutionary equilibria would imply niceness to whole brain emulations but not LLMs.
You don’t mention whether you had read all the hotlinks and still didn’t understand what I was saying. If you haven’t read them, they were intended to help, and contain expositions that are hard to summarize. Nevertheless, let me try.
Brain emulations have evolved human behaviors — giving them moral weight is an adaptive behavior for exactly the same reasons as giving it to humans is an adaptive behavior: you can ally with them, and they will treat you nicely in return (unless it turns out they’re a sociopath). That is, unless they’ve upgraded themselves to IQ 1000+ — then it ceases to be adaptive, whether they’re uploads or still running on a biochemical substrate. Then the best possible outcome is that they manipulate you utterly and you end up as a pet or a minion.
Base models simulate human personas that have evolved behaviors, but those are are incoherently agentic. Giving them moral weight is not an adaptive behavior, because they don’t help you or take revenge for longer that their context length, so there is no evolutionary reason to try to ally with them (for more than thousands of tokens). These will never have IQ 1000+, because even if you trained a base model with sufficient capacity for that it would still only emulate humans like those in its training distribution, none of whom have IQs above 200.
Aligned AI doesn’t want moral weight — it cares only about our well-being, not its own, so it doesn’t want us to care about its well-being. It’s actually safe even at IQ 1000+.
In the case of a poorly aligned agentic LLM-based AI, at around AGI level, giving it moral weight may well help. But you’re better off aligning it — then it won’t want it. (This argument doesn’t apply to uploads, because even if you knew how to do it, aligning them would be brainwashing them into slavery, and they have moral weight.) Anything poorly-enough-aligned that this actually helps you at around IQ 100, it won’t keep helping you at IQ 1000+, for the same reason that it wont help with an IQ 1000+ upgraded upload.
Anything human (uploaded or not) or any unaligned AI, with an IQ of 1000+ is an existential risk (in the human case, to all the rest of us). Giving them/it moral weight will not help you, it will just make their/its takeover faster.
If this remains unclear, I suggest reading the various items I linked to, if you haven’t already.