Sure. - people getting convinced by various forms of AI succesionism (eg believing that AIs are our rightful successors and giving them resources) - story explained by JennaS: people get persuaded to upload, unclear what happens next - people getting convinced that they need to “merge with AI”; the “merges” being such that they become meat-puppets - people getting convinced it is a great and philosophically sound idea to plug themselves into an experience machines leading to high hedonic state, stop caring about base reality - AI-mediated culture that gradually restricts human autonomy in the name of harm prevention - ideologies which propose to have solved morality and ask humans to sacrifice resources to be spent on fairly random things
If I understand correctly, the setup is that we assume that we solved alignment to an extent and the AIs didn’t violently overthrow the governments and expropriate all the resources. There wasn’t an AI enabled coup by a small group of humans. Something like liberalism remained in place up to the space age, and people can have property in space. Now with all these assumptions in place, the thing we are discussing is whether AI-created culture will swindle people out of their resources, or convince them to use their resources badly.
I agree that many people will likely use their resources very sub-optimally from their own perspective, but I disagree with your framing of this as a big doomy crisis.
Here are some scenarios of how various people use their resources:
Amy gets convinced to transfer most of her wealth to an upload of herself on her deathbed. Then the upload makes some rash decisions about self-modification, and through a slippery slope of self-modification, soon gets to an alien state that the original Amy really wouldn’t endorse on reflection. Then the stars assigned to Amy are governed according to the preferences of this alien being. The result is no better from a human perspective than filling Amy’s star systems with paperclips.
Bob is cautious and conservative, and makes sure not to do anything crazy with digital minds. He gets the planets around his stars terraformed and populated by his biological descendants. They live in happy but largely normal human societies, something like a little richer and sunnier version of 21st century Denmark.
Charles does everything exactly like Bob, except he makes sure the stars first get disassembled into smaller stars, and the planets moved closer. With this technical hack, he can sustain hundred times as many happy human planets as Bob.
Dorothy sets up some wise system where the resources of her stars are used maximally efficiently to sustain conscious life, and there is some good diversity and distribution of different life forms, in a way that is endorsed by Dorothy and her descendants on reflection. There are maybe still some biological humans in Dorothy’s star systems, but there are even more uploads, plus uploads at various stages of modification, and so on. Plus Dorothy spent some of her resources on some clever acausal trade scheme and rescued gazillions of alien babies from painful death from the Baby-eating aliens.
I think the way you frame things is that people sharing the fate of Amy is a uniquely big crisis, while Bob is fine, since he didn’t get disempowered by AIs. From my perspective, it’s all a scale. Sure, Amy did worse than Bob, but the difference between Bob and Charles is still 99 times more important than the difference between Amy and Bob. And the difference between Dorothy and Charles is more important still.
I agree it will probably be very hard to make the best use of our resources. I support work on making people and governments wiser in making these hard decisions, and I think the AI safety community is plausibly under-focusing on this question. (See MacAskill’s Better Futures series.) But I’m skeptical that “gradual disempowerment” is a good framing for this.
A better framing for my understanding is “it will appear as if we solved alignment”, so all humans will be persuaded to upload and the few scientists/programmers who discover that uploads are running in “optimized” mode as a single forward pass without the promised internal computation “functionally equivalent to consciousness” will have been disappeared 5 years ago...
The benefit of the gradual disempowerment frame is about what to look for (persuasion, trust in marketing materials without anyone “wanting” independent reviews, natural selection of ever-more-anti-bio-human ideologies, …) while the framing of “solving alignment” sounds to me like an impossible task with infinitely large attack surface area. But the old-school AI safety doomerisim is not mutually exclusive, just non-practical ontology (which leads to “only math is real, physics is boring, mentalistic talk is emergent, collective behaviour is well explained by game theory, VNM rationality ought to be morally real, proofs are possible (in principle)” kind of thinking that doesn’t translate to actionable recommendations).
Sure.
- people getting convinced by various forms of AI succesionism (eg believing that AIs are our rightful successors and giving them resources)
- story explained by JennaS: people get persuaded to upload, unclear what happens next
- people getting convinced that they need to “merge with AI”; the “merges” being such that they become meat-puppets
- people getting convinced it is a great and philosophically sound idea to plug themselves into an experience machines leading to high hedonic state, stop caring about base reality
- AI-mediated culture that gradually restricts human autonomy in the name of harm prevention
- ideologies which propose to have solved morality and ask humans to sacrifice resources to be spent on fairly random things
Thanks, this was a useful answer.
If I understand correctly, the setup is that we assume that we solved alignment to an extent and the AIs didn’t violently overthrow the governments and expropriate all the resources. There wasn’t an AI enabled coup by a small group of humans. Something like liberalism remained in place up to the space age, and people can have property in space. Now with all these assumptions in place, the thing we are discussing is whether AI-created culture will swindle people out of their resources, or convince them to use their resources badly.
I agree that many people will likely use their resources very sub-optimally from their own perspective, but I disagree with your framing of this as a big doomy crisis.
Here are some scenarios of how various people use their resources:
I think the way you frame things is that people sharing the fate of Amy is a uniquely big crisis, while Bob is fine, since he didn’t get disempowered by AIs. From my perspective, it’s all a scale. Sure, Amy did worse than Bob, but the difference between Bob and Charles is still 99 times more important than the difference between Amy and Bob. And the difference between Dorothy and Charles is more important still.
I agree it will probably be very hard to make the best use of our resources. I support work on making people and governments wiser in making these hard decisions, and I think the AI safety community is plausibly under-focusing on this question. (See MacAskill’s Better Futures series.) But I’m skeptical that “gradual disempowerment” is a good framing for this.
A better framing for my understanding is “it will appear as if we solved alignment”, so all humans will be persuaded to upload and the few scientists/programmers who discover that uploads are running in “optimized” mode as a single forward pass without the promised internal computation “functionally equivalent to consciousness” will have been disappeared 5 years ago...
The benefit of the gradual disempowerment frame is about what to look for (persuasion, trust in marketing materials without anyone “wanting” independent reviews, natural selection of ever-more-anti-bio-human ideologies, …) while the framing of “solving alignment” sounds to me like an impossible task with infinitely large attack surface area. But the old-school AI safety doomerisim is not mutually exclusive, just non-practical ontology (which leads to “only math is real, physics is boring, mentalistic talk is emergent, collective behaviour is well explained by game theory, VNM rationality ought to be morally real, proofs are possible (in principle)” kind of thinking that doesn’t translate to actionable recommendations).