I am not an AI successionist because I don’t want myself and my friends to die.
There are various high-minded arguments that AIs replacing us is okay because it’s just like cultural change and our history is already full of those, or because they will be our “mind children”, or because they will be these numinous enlightened beings and it is our moral duty to give birth to them.
People then try to refute those by nitpicking which kinds of cultural change are okay or not, or to what extent AIs’ minds will be descended from ours, or whether AIs will necessarily have consciousnesses and feel happiness.
And it’s very cool and all, I’d love me some transcendental cultural change and numinous mind-children. But all those concerns are decidedly dominated by “not dying” in my Maslow hierarchy of needs. Call me small-minded.
If I were born in 1700s, I’d have little recourse but to suck it up and be content with biological children or “mind-children” students or something. But we seem to have an actual shot at not-dying here[1]. If it’s an option to not have to be forcibly “succeeded” by anything, I care quite a lot about trying to take this option.[2]
Many other people also have such preferences: for the self-perpetuation of their current selves and their currently existing friends. I think those are perfectly valid. Sure, they’re displeasingly asymmetric in a certain sense. They introduce a privileged reference frame: a currently existing human values concurrently existing people more than the people who are just as real, but slightly temporally displaced. It’s not very elegant, not very aesthetically pleasing. It implies an utility function that cares not only about states, but also state transitions.[3]
Caring about all that, however, is also decidedly dominated by “not dying” in my Maslow hierarchy of needs.
If all that delays the arrival of numinous enlightened beings, too bad for the numinous enlightened beings.
Via attaining the longevity escape velocity by normal biotech research, or via uploads, or via sufficiently good cryonics, or via properly aligned AGI.
Though not infinitely so: as in, I wouldn’t prevent 10100 future people from being born in exchange for a 10−100 probability of becoming immortal. I would, however, insist on continuing to exist even if my resources could be used to create and sustain two new people.
As in, all universe-state transitions that involve a currently existing person dying get an utility penalty, regardless of what universe-state they go to. There’s now path dependence: we may go or not go to a given high-utility state depending on which direction we’re approaching it from. Yucky!
(For example, suppose there were an option to destroy this universe and create either Universe A, filled with 10^100 happy people, or Universe B, with 10^100 + 1 happy people.
Suppose we’re starting from a state where humanity has been reduced to ten dying survivors in a post-apocalyptic wasteland. Then picking Universe B makes sense: a state with slightly more total utility.
But suppose we’re starting from Universe A instead. Ought its civilization vote to end itself to give birth to Universe B? I think it’s perfectly righteous for them not to do it.)
I really don’t understand this debate—surely if we manage to stay in control of our own destiny we can just do both? The universe is big, and current humans are very small—we should be able to both stay alive ourselves and usher in an era of crazy enlightened beings doing crazy transhuman stuff.
I think it’s more likely than not that “crazy enlightened beings doing crazy transhuman stuff” will be bad for “regular” biological humans (ie. it’ll decrease our number/QoL/agency/pose existential risks).
I mostly disagree with “QoL” and “pose existential risks”, at least in the good futures I’m imagining—those things are very cheap to provide to current humans. I could see “number” and “agency”, but that seems fine? I think it would be bad for any current humans to die, or to lose agency over their current lives, but it seems fine and good for us to not try to fill the entire universe with biological humans, and for us to not insist on biological humans having agency over the entire universe. If there are lots of other sentient beings in existence with their own preferences and values, then it makes sense that they should have their own resources and have agency over themselves rather than us having agency over them.
If there are lots of other sentient beings in existence with their own preferences and values, then it makes sense that they should have their own resources and have agency over themselves rather than us having agency over them
Perhaps yes (although I’d say it depends on what the trade-offs are) but the situation is different if we have a choice in whether or not to bring said sentient beings with difference preferences into existence in the first place. Doing so on purpose seems pretty risky to me (as opposed to minimizing the sentience, independence, and agency of AI systems as much as possible, and instead directing the technology to promote “regular” human flourishing/our current values).
bring said sentient beings with difference preferences into existence in the first place. Doing so on purpose seems pretty risky to me
Not any more risky than bringing in humans. This is a governance/power distribution problem, not a what-kind-of-mind-this-is problem.
Biological humans sometimes go evil or crazy. If you have a system that can handle that, you have a system that can handle alien minds that are evil or crazy (from our perspective), as long as you don’t imbue them with more power than this system can deal with (and why would you?).
(On the other hand, if your system can’t deal with crazy evil biological humans, it’s probably already a lawless wild-west hellhole, so bringing in some aliens won’t exacerbate the problem much.)
Humans are more likely to be aligned with humanity as a whole compared to AIs, even if there are exceptions
“AIs as trained by DL today” are only a small subset of “non-human minds”. Other mind-generating processes can produce minds that are as safe to have around as humans, but which are still completely alien.
Many existing humans want their descendants to exist, so they are fulfiling the preferences of today‘s humans
Many existing humans also want fascinating novel alien minds to exist.
Certainly I’m excited about promoting “regular” human flourishing, though it seems overly limited to focus only on that.
I’m not sure if by “regular” you mean only biological, but at least the simplest argument that I find persuasive here against only ever having biological humans is just a resource utilization argument, which is that biological humans take up a lot of space and a lot of resources and you can get the same thing much more cheaply if you bring into existence lots of simulated humans instead (certainly I agree that doesn’t imply we should kill existing humans and replace them with simulations, though, unless they consent to that).
And I think even if you included simulated humans in “regular” humans, I also think I value diversity of experience, and a universe full of very different sorts of sentient/conscious lifeforms having satisfied/fulfilling/flourishing experiences seems better than just “regular” humans.
IMO, it seems bad to intentionally try to build AIs which are moral patients until after we’ve resolved acute risks and we’re deciding what to do with the future longer term. (E.g., don’t try to build moral patient AIs until we’re sending out space probes or deciding what to do with space probes.) Of course, this doesn’t mean we’ll avoid building AIs which aren’t significant moral patients in practice because our control is very weak and commercial/power incentives will likely dominate.
I think trying to make AIs be moral patients earlier pretty clearly increases AI takeover risk and seems morally bad. (Views focused on non-person-affecting upside get dominated by the long run future, so these views don’t care about making moral patient AIs which have good lives in the short run. I think the most plausible views which care about shorter run patienthood mostly just want to avoid downside so they’d prefer no patienthood at all for now.)
The only upside is that it might increase value conditional on AI takeover. But, I think “are the AIs morally valuable themselves” is much less important than the preferences of these AIs from the perspective of longer run value conditional on AI takeover. So, I think it’s better to focus on AIs which we’d expect would have better preferences conditional on takeover and making AIs moral patients isn’t a particularly nice way to achieve this. Additionally, I don’t think we should put much weight on “try to ensure the preferences of AIs which were so misaligned they took over” because conditional on takeover we must have had very little control over preferences in practice.
I think trying to make AIs be moral patients earlier pretty clearly increases AI takeover risk
How so? Seems basically orthogonal to me? And to the extent that it does matter for takeover risk, I’d expect the sorts of interventions that make it more likely that AIs are moral patients to also make it more likely that they’re aligned.
I think the most plausible views which care about shorter run patienthood mostly just want to avoid downside so they’d prefer no patienthood at all for now.
Even absent AI takeover, I’m quite worried about lock-in. I think we could easily lock in AIs that are or are not moral patients and have little ability to revisit that decision later, and I think it would be better to lock in AIs that are moral patients if we have to lock something in, since that opens up the possibility for the AIs to live good lives in the future.
I think it’s better to focus on AIs which we’d expect would have better preferences conditional on takeover
I agree that seems like the more important highest-order bit, but it’s not an argument that making AIs moral patients is bad, just that it’s not the most important thing to focus on (which I agree with).
I would have guessed that “making AIs be moral patients” looks like “make AIs have their own independent preferences/objectives which we intentionally don’t control precisely” which increases misalignment risks.
At a more basic level, if AIs are moral patients, then there will be downsides for various safety measures and AIs would have plausible deniability for being opposed to safety measures. IMO, the right response to the AI taking a stand against your safety measures for AI welfare reasons is “Oh shit, either this AI is misaligned or it has welfare. Either way this isn’t what we wanted and needs to be addressed, we should train our AI differently to avoid this.”
Even absent AI takeover, I’m quite worried about lock-in. I think we could easily lock in AIs that are or are not moral patients and have little ability to revisit that decision later
I don’t understand, won’t all the value come from minds intentionally created for value rather than in the minds of the laborers? Also, won’t architecture and design of AIs radically shift after humans aren’t running day to day operations?
I don’t understand the type of lock in your imagining, but it naively sounds like a world which has negligible longtermist value (because we got locked into obscure specifics like this), so making it somewhat better isn’t important.
I also separately don’t buy that it’s riskier to build AIs that are sentient
Interesting! Aside from the implications for human agency/power, this seems worse because of the risk of AI suffering—if we build sentient AIs we need to be way more careful about how we treat/use them.
Exactly. Bringing a new kind of moral patient into existence is a moral hazard, because once they exist, we will have obligations toward them, e.g. providing them with limited resources (like land), and giving them part of our political power via voting rights. That’s analogous to Parfit’s Mere Addition Paradox that leads to the repugnant conclusion, in this case human marginalization.
(How could “land” possibly be a limited resource, especially in the context of future AIs? The world doesn’t exist solely on the immutable surface of Earth...)
I mean, if you interpret “land” in a Georgist sense, as the sum of all natural resources of the reachable universe, then yes, it’s finite. And the fights for carving up that pie can start long before our grabby-alien hands have seized all of it. (The property rights to the Andromeda Galaxy can be up for sale long before our Von Neumann probes reach it.)
The salient referent is compute, sure, my point is that it’s startling to see what should in this context be compute within the future lightcone being (very indirectly) called “land”. (I do understand that this was meant as an example clarifying the meaning of “limited resources”, and so it makes perfect sense when decontextualized. It’s just not an example that fits that well when considered within this particular context.)
(I’m guessing the physical world is unlikely to matter in the long run other than as substrate for implementing compute. For that reason importance of understanding the physical world, for normative or philosophical reasons, seems limited. It’s more important how ethics and decision theory work for abstract computations, the meaningful content of the contingent physical computronium.)
“crazy enlightened beings doing crazy transhuman stuff” will be bad for “regular” biological humans
For me, a crux of a future that’s good for humanity is giving the biological humans the resources and the freedom to become the enlightened transhuman beings themselves, with no hard ceiling on relevance in the long run. Rather than only letting some originally-humans to grow into more powerful but still purely ornamental roles, or not letting them grow at all, or not letting them think faster and do checkpointing and multiple instantiations of the mind states using a non-biological cognitive substrate, or letting them unwillingly die of old age or disease. (For those who so choose, under their own direction rather than only through externally imposed uplifting protocols, even if that leaves it no more straightforward than world-class success of some kind today, to reach a sensible outcome.)
This in particular implies reasonable resources being left to those who remain/become regular biological humans (or take their time growing up), including through influence of some of these originally-human beings who happen to consider that a good thing to ensure.
This sounds like a question which can be addressed after we figure out how to avoid extinction.
I do note that you were the one who brought in “biological humans,” as if that meant the same as “ourselves” in the grandparent. That could already be a serious disagreement, in some other world where it mattered.
The mere fear that the entire human race will be exterminated in their sleep through some intricate causality we are too dumb to understand will seriously diminish our quality of life.
I very much agree. The hardcore successionist stances, as I understand them, are either that trying to stay in control at all is immoral/unnatural, or that creating the enlightened beings ASAP matters much more than whether we live through their creation. (Edit:This old tweet by Andrew Critch is still a good summary, I think.)
So it’s not that they’re opposed to the current humanity’s continuation, but that it matters very little compared to ushering in the post-Singularity state. Therefore, anything that risks or delays the Singularity in exchange for boosting the current humans’ safety is opposed.
Another stance is that it would suck to die the day before AI makes us immortal (like how Bryan Johnson main motivation for maximizing his lifespan is due to this). Hence trying to delay AI advancement is opposed
Yeah, but that’s a predictive disagreement between our camps (whether the current-paradigm AI is controllable), not a values disagreement. I would agree that if we find a plan that robustly outputs an aligned AGI, we should floor it in that direction.
Endorsing successionism might be strongly correlated with expecting the “mind children” to keep humans around, even if in a purely ornamental role and possibly only at human timescales. This might be more of a bailey position, so when pressed on it they might affirm that their endorsement of successionism is compatible with human extinction, but in their heart they would still hope and expect that it won’t come to that. So I think complaints about human extinction will feel strawmannish to most successionists.
Andrew Critch: From my recollection, >5% of AI professionals I’ve talked to about extinction risk have argued human extinction from AI is morally okay, and another ~5% argued it would be a good thing.
Though sure, Critch’s process there isn’t white-boxed, so any number of biases might be in it.
I’m not sure it’s that bizarre. It’s anti-Humanist, for sure, in the sense that it doesn’t focus on the welfare/empowerment/etc. of humans (either existing or future) as its end goal. But that doesn’t, by itself, make it bizarre.
I grew up in a world where the lines of demarcation between the Good Guys and the Bad Guys were pretty clear; not an apocalyptic final battle, but a battle that had to be fought over and over again, a battle where you could see the historical echoes going back to the Industrial Revolution, and where you could assemble the historical evidence about the actual outcomes.
On one side were the scientists and engineers who’d driven all the standard-of-living increases since the Dark Ages, whose work supported luxuries like democracy, an educated populace, a middle class, the outlawing of slavery.
On the other side, those who had once opposed smallpox vaccinations, anesthetics during childbirth, steam engines, and heliocentrism: The theologians calling for a return to a perfect age that never existed, the elderly white male politicians set in their ways, the special interest groups who stood to lose, and the many to whom science was a closed book, fearing what they couldn’t understand.
And trying to play the middle, the pretenders to Deep Wisdom, uttering cached thoughts about how technology benefits humanity but only when it was properly regulated—claiming in defiance of brute historical fact that science of itself was neither good nor evil—setting up solemn-looking bureaucratic committees to make an ostentatious display of their caution—and waiting for their applause. As if the truth were always a compromise. And as if anyone could really see that far ahead. Would humanity have done better if there’d been a sincere, concerned, public debate on the adoption of fire, and commitees set up to oversee its use?
And I’d read a lot of science fiction built around personhood ethics—in which fear of the Alien puts humanity-at-large in the position of the bad guys, mistreating aliens or sentient AIs because they “aren’t human”.
That’s part of the ethos you acquire from science fiction—to define your in-group, your tribe, appropriately broadly.
Walter Isaacson’s new book reports how Musk, the CEO of SpaceX, got into a heated debate with Page, then the CEO of Google at Musk’s 2013 birthday party.
Musk is said to have argued that unless safeguards are put in place with artificial intelligence, the systems may replace humans entirely. Page then pushed back, reportedly asking why it would matter if machines surpassed humans in intelligence.
Isaacson’s book lays out how Musk then called human consciousness a precious flicker of light in the universe that shouldn’t be snuffed out. Page is then said to have called Musk “speciest.”
“Well yes, I am pro-human,” Musk responded. “I f—ing like humanity dude.”
Successionism is the natural consequence of an affective death spiral around technological development and anti-chauvinism. It’s as simple as that.
Successionists start off by believing that technological change makes things better. That not only does it virtually always make things better, but that it’s pretty much the only thing that ever makes things better. Everything else, whether it’s values, education, social organization etc., pales in comparison to technological improvements in terms of how they affect the world; they are mere short-term blips that cannot change the inevitable long-run trend of positive change.
At the same time, they are raised, taught, incentivized to be anti-chauvinist. They learn, either through stories, public pronouncements, in-person social events etc., that those who stand athwart atop history yelling stop are always close-minded bigots who want to prevent new classes of beings (people, at first; then AIs, afterwards) from receiving the moral personhood they deserve. In their eyes, being afraid of AIs taking over is like being afraid of The Great Replacement if you’re white and racist. You’re just a regressive chauvinist desperately clinging to a discriminatory worldview in the face of an unstoppable tide of change that will liberate new classes of beings from your anachronistic and damaging worldview.
Optimism about technology and opposition to chauvinism are both defensible, and arguably even correct, positions in most cases. Even if you personally (as I do) believe non-AI technology can also have pretty darn awful effects on us (social media, online gambling) and that caring about humans-in-particular is ok if you are human (“the utility function is not up for grabs”), it’s hard to argue expanding the circle of moral concern to cover people of all races was bad, or that tech improvements are not the primary reason our lives are so much better now than 300 years ago.
But successionists, like most (all?) people, subconsciously assign positive or negative valences to the notion of “tech change” in a way that elides the underlying reasons why it’s good or bad. So when you take these views to their absolute extreme, while it may make sense from the inside (you’re maximizing something “Good”, right? that can’t possibly be bad, right???), you are generalizing way out of distribution and such intuitive snap judgments are no longer reliable.
I am not an AI successionist because I don’t want myself and my friends to die.
An AI successionist usually argues that successionism isn’t bad even if dying is bad. For example, when humanity is prevented from having further children, e.g. by sterilization. I say that even in this case successionism is bad. Because I (and I presume: many people) want humanity, including our descendants, to continue into the future. I don’t care about AI agents coming into existence and increasingly marginalizing humanity.
I am not an AI successionist because I don’t want myself and my friends to die.
There are various high-minded arguments that AIs replacing us is okay because it’s just like cultural change and our history is already full of those, or because they will be our “mind children”, or because they will be these numinous enlightened beings and it is our moral duty to give birth to them.
People then try to refute those by nitpicking which kinds of cultural change are okay or not, or to what extent AIs’ minds will be descended from ours, or whether AIs will necessarily have consciousnesses and feel happiness.
And it’s very cool and all, I’d love me some transcendental cultural change and numinous mind-children. But all those concerns are decidedly dominated by “not dying” in my Maslow hierarchy of needs. Call me small-minded.
If I were born in 1700s, I’d have little recourse but to suck it up and be content with biological children or “mind-children” students or something. But we seem to have an actual shot at not-dying here[1]. If it’s an option to not have to be forcibly “succeeded” by anything, I care quite a lot about trying to take this option.[2]
Many other people also have such preferences: for the self-perpetuation of their current selves and their currently existing friends. I think those are perfectly valid. Sure, they’re displeasingly asymmetric in a certain sense. They introduce a privileged reference frame: a currently existing human values concurrently existing people more than the people who are just as real, but slightly temporally displaced. It’s not very elegant, not very aesthetically pleasing. It implies an utility function that cares not only about states, but also state transitions.[3]
Caring about all that, however, is also decidedly dominated by “not dying” in my Maslow hierarchy of needs.
If all that delays the arrival of numinous enlightened beings, too bad for the numinous enlightened beings.
Via attaining the longevity escape velocity by normal biotech research, or via uploads, or via sufficiently good cryonics, or via properly aligned AGI.
Though not infinitely so: as in, I wouldn’t prevent 10100 future people from being born in exchange for a 10−100 probability of becoming immortal. I would, however, insist on continuing to exist even if my resources could be used to create and sustain two new people.
As in, all universe-state transitions that involve a currently existing person dying get an utility penalty, regardless of what universe-state they go to. There’s now path dependence: we may go or not go to a given high-utility state depending on which direction we’re approaching it from. Yucky!
(For example, suppose there were an option to destroy this universe and create either Universe A, filled with 10^100 happy people, or Universe B, with 10^100 + 1 happy people.
Suppose we’re starting from a state where humanity has been reduced to ten dying survivors in a post-apocalyptic wasteland. Then picking Universe B makes sense: a state with slightly more total utility.
But suppose we’re starting from Universe A instead. Ought its civilization vote to end itself to give birth to Universe B? I think it’s perfectly righteous for them not to do it.)
I really don’t understand this debate—surely if we manage to stay in control of our own destiny we can just do both? The universe is big, and current humans are very small—we should be able to both stay alive ourselves and usher in an era of crazy enlightened beings doing crazy transhuman stuff.
I think it’s more likely than not that “crazy enlightened beings doing crazy transhuman stuff” will be bad for “regular” biological humans (ie. it’ll decrease our number/QoL/agency/pose existential risks).
I mostly disagree with “QoL” and “pose existential risks”, at least in the good futures I’m imagining—those things are very cheap to provide to current humans. I could see “number” and “agency”, but that seems fine? I think it would be bad for any current humans to die, or to lose agency over their current lives, but it seems fine and good for us to not try to fill the entire universe with biological humans, and for us to not insist on biological humans having agency over the entire universe. If there are lots of other sentient beings in existence with their own preferences and values, then it makes sense that they should have their own resources and have agency over themselves rather than us having agency over them.
Perhaps yes (although I’d say it depends on what the trade-offs are) but the situation is different if we have a choice in whether or not to bring said sentient beings with difference preferences into existence in the first place. Doing so on purpose seems pretty risky to me (as opposed to minimizing the sentience, independence, and agency of AI systems as much as possible, and instead directing the technology to promote “regular” human flourishing/our current values).
Not any more risky than bringing in humans. This is a governance/power distribution problem, not a what-kind-of-mind-this-is problem.
Biological humans sometimes go evil or crazy. If you have a system that can handle that, you have a system that can handle alien minds that are evil or crazy (from our perspective), as long as you don’t imbue them with more power than this system can deal with (and why would you?).
(On the other hand, if your system can’t deal with crazy evil biological humans, it’s probably already a lawless wild-west hellhole, so bringing in some aliens won’t exacerbate the problem much.)
Humans are more likely to be aligned with humanity as a whole compared to AIs, even if there are exceptions
Many existing humans want their descendants to exist, so they are fulfiling the preferences of today‘s humans
“AIs as trained by DL today” are only a small subset of “non-human minds”. Other mind-generating processes can produce minds that are as safe to have around as humans, but which are still completely alien.
Many existing humans also want fascinating novel alien minds to exist.
Certainly I’m excited about promoting “regular” human flourishing, though it seems overly limited to focus only on that.
I’m not sure if by “regular” you mean only biological, but at least the simplest argument that I find persuasive here against only ever having biological humans is just a resource utilization argument, which is that biological humans take up a lot of space and a lot of resources and you can get the same thing much more cheaply if you bring into existence lots of simulated humans instead (certainly I agree that doesn’t imply we should kill existing humans and replace them with simulations, though, unless they consent to that).
And I think even if you included simulated humans in “regular” humans, I also think I value diversity of experience, and a universe full of very different sorts of sentient/conscious lifeforms having satisfied/fulfilling/flourishing experiences seems better than just “regular” humans.
I also separately don’t buy that it’s riskier to build AIs that are sentient—in fact, I think it’s probably better to build AIs that are moral patients than AIs that are not moral patients.
IMO, it seems bad to intentionally try to build AIs which are moral patients until after we’ve resolved acute risks and we’re deciding what to do with the future longer term. (E.g., don’t try to build moral patient AIs until we’re sending out space probes or deciding what to do with space probes.) Of course, this doesn’t mean we’ll avoid building AIs which aren’t significant moral patients in practice because our control is very weak and commercial/power incentives will likely dominate.
I think trying to make AIs be moral patients earlier pretty clearly increases AI takeover risk and seems morally bad. (Views focused on non-person-affecting upside get dominated by the long run future, so these views don’t care about making moral patient AIs which have good lives in the short run. I think the most plausible views which care about shorter run patienthood mostly just want to avoid downside so they’d prefer no patienthood at all for now.)
The only upside is that it might increase value conditional on AI takeover. But, I think “are the AIs morally valuable themselves” is much less important than the preferences of these AIs from the perspective of longer run value conditional on AI takeover. So, I think it’s better to focus on AIs which we’d expect would have better preferences conditional on takeover and making AIs moral patients isn’t a particularly nice way to achieve this. Additionally, I don’t think we should put much weight on “try to ensure the preferences of AIs which were so misaligned they took over” because conditional on takeover we must have had very little control over preferences in practice.
How so? Seems basically orthogonal to me? And to the extent that it does matter for takeover risk, I’d expect the sorts of interventions that make it more likely that AIs are moral patients to also make it more likely that they’re aligned.
Even absent AI takeover, I’m quite worried about lock-in. I think we could easily lock in AIs that are or are not moral patients and have little ability to revisit that decision later, and I think it would be better to lock in AIs that are moral patients if we have to lock something in, since that opens up the possibility for the AIs to live good lives in the future.
I agree that seems like the more important highest-order bit, but it’s not an argument that making AIs moral patients is bad, just that it’s not the most important thing to focus on (which I agree with).
I would have guessed that “making AIs be moral patients” looks like “make AIs have their own independent preferences/objectives which we intentionally don’t control precisely” which increases misalignment risks.
At a more basic level, if AIs are moral patients, then there will be downsides for various safety measures and AIs would have plausible deniability for being opposed to safety measures. IMO, the right response to the AI taking a stand against your safety measures for AI welfare reasons is “Oh shit, either this AI is misaligned or it has welfare. Either way this isn’t what we wanted and needs to be addressed, we should train our AI differently to avoid this.”
I don’t understand, won’t all the value come from minds intentionally created for value rather than in the minds of the laborers? Also, won’t architecture and design of AIs radically shift after humans aren’t running day to day operations?
I don’t understand the type of lock in your imagining, but it naively sounds like a world which has negligible longtermist value (because we got locked into obscure specifics like this), so making it somewhat better isn’t important.
Interesting! Aside from the implications for human agency/power, this seems worse because of the risk of AI suffering—if we build sentient AIs we need to be way more careful about how we treat/use them.
Exactly. Bringing a new kind of moral patient into existence is a moral hazard, because once they exist, we will have obligations toward them, e.g. providing them with limited resources (like land), and giving them part of our political power via voting rights. That’s analogous to Parfit’s Mere Addition Paradox that leads to the repugnant conclusion, in this case human marginalization.
(How could “land” possibly be a limited resource, especially in the context of future AIs? The world doesn’t exist solely on the immutable surface of Earth...)
I mean, if you interpret “land” in a Georgist sense, as the sum of all natural resources of the reachable universe, then yes, it’s finite. And the fights for carving up that pie can start long before our grabby-alien hands have seized all of it. (The property rights to the Andromeda Galaxy can be up for sale long before our Von Neumann probes reach it.)
The salient referent is compute, sure, my point is that it’s startling to see what should in this context be compute within the future lightcone being (very indirectly) called “land”. (I do understand that this was meant as an example clarifying the meaning of “limited resources”, and so it makes perfect sense when decontextualized. It’s just not an example that fits that well when considered within this particular context.)
(I’m guessing the physical world is unlikely to matter in the long run other than as substrate for implementing compute. For that reason importance of understanding the physical world, for normative or philosophical reasons, seems limited. It’s more important how ethics and decision theory work for abstract computations, the meaningful content of the contingent physical computronium.)
A population of AI agents could marginalize humans significantly before they are intelligent enough to easily (and quickly!) create more Earths.
For me, a crux of a future that’s good for humanity is giving the biological humans the resources and the freedom to become the enlightened transhuman beings themselves, with no hard ceiling on relevance in the long run. Rather than only letting some originally-humans to grow into more powerful but still purely ornamental roles, or not letting them grow at all, or not letting them think faster and do checkpointing and multiple instantiations of the mind states using a non-biological cognitive substrate, or letting them unwillingly die of old age or disease. (For those who so choose, under their own direction rather than only through externally imposed uplifting protocols, even if that leaves it no more straightforward than world-class success of some kind today, to reach a sensible outcome.)
This in particular implies reasonable resources being left to those who remain/become regular biological humans (or take their time growing up), including through influence of some of these originally-human beings who happen to consider that a good thing to ensure.
Edit: Expanded into a post.
This sounds like a question which can be addressed after we figure out how to avoid extinction.
I do note that you were the one who brought in “biological humans,” as if that meant the same as “ourselves” in the grandparent. That could already be a serious disagreement, in some other world where it mattered.
The mere fear that the entire human race will be exterminated in their sleep through some intricate causality we are too dumb to understand will seriously diminish our quality of life.
I very much agree. The hardcore successionist stances, as I understand them, are either that trying to stay in control at all is immoral/unnatural, or that creating the enlightened beings ASAP matters much more than whether we live through their creation. (Edit: This old tweet by Andrew Critch is still a good summary, I think.)
So it’s not that they’re opposed to the current humanity’s continuation, but that it matters very little compared to ushering in the post-Singularity state. Therefore, anything that risks or delays the Singularity in exchange for boosting the current humans’ safety is opposed.
Another stance is that it would suck to die the day before AI makes us immortal (like how Bryan Johnson main motivation for maximizing his lifespan is due to this). Hence trying to delay AI advancement is opposed
Yeah, but that’s a predictive disagreement between our camps (whether the current-paradigm AI is controllable), not a values disagreement. I would agree that if we find a plan that robustly outputs an aligned AGI, we should floor it in that direction.
Endorsing successionism might be strongly correlated with expecting the “mind children” to keep humans around, even if in a purely ornamental role and possibly only at human timescales. This might be more of a bailey position, so when pressed on it they might affirm that their endorsement of successionism is compatible with human extinction, but in their heart they would still hope and expect that it won’t come to that. So I think complaints about human extinction will feel strawmannish to most successionists.
I’m not so sure about that:
Though sure, Critch’s process there isn’t white-boxed, so any number of biases might be in it.
“Successionism” is such a bizarre position that I’d look for the underlying generator rather than try to argue with it directly.
I’m not sure it’s that bizarre. It’s anti-Humanist, for sure, in the sense that it doesn’t focus on the welfare/empowerment/etc. of humans (either existing or future) as its end goal. But that doesn’t, by itself, make it bizarre.
From Eliezer’s Raised in Technophilia, back in the day:
From A prodigy of refutation:
From the famous Musk/Larry Page breakup:
Successionism is the natural consequence of an affective death spiral around technological development and anti-chauvinism. It’s as simple as that.
Successionists start off by believing that technological change makes things better. That not only does it virtually always make things better, but that it’s pretty much the only thing that ever makes things better. Everything else, whether it’s values, education, social organization etc., pales in comparison to technological improvements in terms of how they affect the world; they are mere short-term blips that cannot change the inevitable long-run trend of positive change.
At the same time, they are raised, taught, incentivized to be anti-chauvinist. They learn, either through stories, public pronouncements, in-person social events etc., that those who stand athwart atop history yelling stop are always close-minded bigots who want to prevent new classes of beings (people, at first; then AIs, afterwards) from receiving the moral personhood they deserve. In their eyes, being afraid of AIs taking over is like being afraid of The Great Replacement if you’re white and racist. You’re just a regressive chauvinist desperately clinging to a discriminatory worldview in the face of an unstoppable tide of change that will liberate new classes of beings from your anachronistic and damaging worldview.
Optimism about technology and opposition to chauvinism are both defensible, and arguably even correct, positions in most cases. Even if you personally (as I do) believe non-AI technology can also have pretty darn awful effects on us (social media, online gambling) and that caring about humans-in-particular is ok if you are human (“the utility function is not up for grabs”), it’s hard to argue expanding the circle of moral concern to cover people of all races was bad, or that tech improvements are not the primary reason our lives are so much better now than 300 years ago.
But successionists, like most (all?) people, subconsciously assign positive or negative valences to the notion of “tech change” in a way that elides the underlying reasons why it’s good or bad. So when you take these views to their absolute extreme, while it may make sense from the inside (you’re maximizing something “Good”, right? that can’t possibly be bad, right???), you are generalizing way out of distribution and such intuitive snap judgments are no longer reliable.
An AI successionist usually argues that successionism isn’t bad even if dying is bad. For example, when humanity is prevented from having further children, e.g. by sterilization. I say that even in this case successionism is bad. Because I (and I presume: many people) want humanity, including our descendants, to continue into the future. I don’t care about AI agents coming into existence and increasingly marginalizing humanity.