where the future of humanity only gets a tiny sliver of the reachable universe
I am not sure how to think about this. “Canned primates” are not going to reach a big part of the physically reachable universe. For the purposes of thinking about “the light cone”, one should still think about “merge with AI”, “uploading”, and so on. That line of reasoning should be not about “humans vs AIs”, but about ways to have a “good merge” (that is, without succumbing to S-risks, and without doing bad things to unmodified biologicals).
Also, I tend to privilege already living humans or their close descendants over the more remote ones, so achieving personal immortality is important if one wants to enjoy a sizable chunk of “the light cone” (it takes time to reach it). Of course, we need personal immortality ASAP anyway, otherwise their “everyones dies” would really become true (although not all at once, and not without replacement, but that’s cold comfort for those currently alive).
That’s the intended meaning, I go into more detail in the linked post. Hence “the future of humanity” rather than simply “humanity”, something humanity would endorse as its future, which is not exclusively (or at all) biological humans. Currently living humans could in principle develop tools to uplift themselves all the way to star-sized superintelligences, but that requires a star, while what humans might instead get is a metaphorical server rack, hence permanent disempowerment.
My comment is primarily an objection about vague terminology not distinguishing permanent disempowerment from extinction. Avoiding permanent disempowerment seems like the correct shared cause, while the cause of merely non-extinction has many ways of endorsing plans that lead to permanent disempowerment. And not being content with permanent disempowerment (even under the conditions of eutopia within strict constraints on resources) depends on noticing that more is possible.
What I am going to say is semi-off-topic for this post (I was trying not to consider potential object-level disagreements), but I have noticed that when discussing human intelligence augmentation, the authors of IABIED always talk only about genetic enhancements and never about direct merge between humans and electronic devices (which seems to also be consistent with their past writings on this). So it seems that for unspecified (but perhaps very rational) reasons, they want to keep enhanced humans purely biological for quite a while.
(Perhaps they think that we can’t handle close coupling of humans and electronics in a way which is existentially safe at this time.)
Whereas, sufficient uplifting requires fairly radical changes. And, in any case, intelligence augmentation via coupling with electronics is likely to be a much faster path and to produce a more radical intelligence augmentation. But, perhaps, they think that the associated existential risks are too high...
Genetic enhancement seems like a safe-ish way of getting a few standard deviations without yet knowing what you are really doing, that current humanity could actually attempt in practice. And that might help a lot with both the “knowing what you are doing” part, and with not doing irreversible things without knowing what you are doing. Any change risks misalignment, uplifting to a superintelligence requires ASI-grade alignment theory and technology, even lifespans for baseline biological humans that run into centuries risk misalignment (since this never happened before). There’s always cryonics, which enables waiting for future progress, if civilization was at all serious about it.
So when you talk about “merging with AI”, that is very suspicious, because a well-developed uplifting methodology doesn’t obviously look anything like “merging with AI”. You become some kind of more capable mind, that’s different from what you were before, not taking irreversible steps towards something you wouldn’t endorse. Without such a methodology, it’s a priori about as bad an idea as building superintelligence in 2029.
I usually think about “reversible merges” for the purpose of intelligence augmentation (not for the purpose of space travel, though).
I tend to think that high-end non-invasive BCI are powerful enough for that and safer than implants. But yes, there still might be serious risks, both personal and existential.
I am not sure how to think about this. “Canned primates” are not going to reach a big part of the physically reachable universe. For the purposes of thinking about “the light cone”, one should still think about “merge with AI”, “uploading”, and so on. That line of reasoning should be not about “humans vs AIs”, but about ways to have a “good merge” (that is, without succumbing to S-risks, and without doing bad things to unmodified biologicals).
Also, I tend to privilege already living humans or their close descendants over the more remote ones, so achieving personal immortality is important if one wants to enjoy a sizable chunk of “the light cone” (it takes time to reach it). Of course, we need personal immortality ASAP anyway, otherwise their “everyones dies” would really become true (although not all at once, and not without replacement, but that’s cold comfort for those currently alive).
That’s the intended meaning, I go into more detail in the linked post. Hence “the future of humanity” rather than simply “humanity”, something humanity would endorse as its future, which is not exclusively (or at all) biological humans. Currently living humans could in principle develop tools to uplift themselves all the way to star-sized superintelligences, but that requires a star, while what humans might instead get is a metaphorical server rack, hence permanent disempowerment.
My comment is primarily an objection about vague terminology not distinguishing permanent disempowerment from extinction. Avoiding permanent disempowerment seems like the correct shared cause, while the cause of merely non-extinction has many ways of endorsing plans that lead to permanent disempowerment. And not being content with permanent disempowerment (even under the conditions of eutopia within strict constraints on resources) depends on noticing that more is possible.
Yes.
What I am going to say is semi-off-topic for this post (I was trying not to consider potential object-level disagreements), but I have noticed that when discussing human intelligence augmentation, the authors of IABIED always talk only about genetic enhancements and never about direct merge between humans and electronic devices (which seems to also be consistent with their past writings on this). So it seems that for unspecified (but perhaps very rational) reasons, they want to keep enhanced humans purely biological for quite a while.
(Perhaps they think that we can’t handle close coupling of humans and electronics in a way which is existentially safe at this time.)
Whereas, sufficient uplifting requires fairly radical changes. And, in any case, intelligence augmentation via coupling with electronics is likely to be a much faster path and to produce a more radical intelligence augmentation. But, perhaps, they think that the associated existential risks are too high...
Genetic enhancement seems like a safe-ish way of getting a few standard deviations without yet knowing what you are really doing, that current humanity could actually attempt in practice. And that might help a lot with both the “knowing what you are doing” part, and with not doing irreversible things without knowing what you are doing. Any change risks misalignment, uplifting to a superintelligence requires ASI-grade alignment theory and technology, even lifespans for baseline biological humans that run into centuries risk misalignment (since this never happened before). There’s always cryonics, which enables waiting for future progress, if civilization was at all serious about it.
So when you talk about “merging with AI”, that is very suspicious, because a well-developed uplifting methodology doesn’t obviously look anything like “merging with AI”. You become some kind of more capable mind, that’s different from what you were before, not taking irreversible steps towards something you wouldn’t endorse. Without such a methodology, it’s a priori about as bad an idea as building superintelligence in 2029.
I usually think about “reversible merges” for the purpose of intelligence augmentation (not for the purpose of space travel, though).
I tend to think that high-end non-invasive BCI are powerful enough for that and safer than implants. But yes, there still might be serious risks, both personal and existential.