First, this presupposes that for any amount of suffering there is some amount of pleasure/bliss/happiness/eudaimonia that could outweigh it. Not all LWers accept this, so it’s worth pointing that out.
But I don’t think the eternal paradise/mediocrity/hell scenario accurately represents what is likely to happen in that scenario. I’d be more worried about somebody using AGI to conquer the world and establish a stable totalitarian system built on some illiberal system, like shariah (according to Caplan, it’s totally plausible for global totalitarianism to persist indefinitely). If you get to post-scarcity, you may grant all your subjects UBI, all basic needs met, etc. (or you may not, if you decide that this policy contradicts Quran or hadith), but if your convictions are strong enough, women will still be forced to wear burkas, be basically slaves of their male kin etc. One could make an argument that abundance robustly promotes more liberal worldview, loosening of social norms, etc., but AFAIK there is no robust evidence for that.
This is meant just to illustrate that you don’t need an outgroup to impose a lot of suffering. Having a screwed up normative framework is just enough.
This family of scenarios is probably still better than AGI doom though.
I kept thinking how a theocracy, assuming it did adopt all the advanced technology we are almost certain is possible but lack the ability to implement, could deal with all these challenges to it’s beliefs.
Half the population is being mistreated because they were born the wrong gender/the wrong ethnic subgroup? No problem, they’ll just go gender transition to the favored group. Total body replacement would be possible so there would be no way to tell who did this.
Sure the theocracy could ban the treatments but it conquered the solar system—it had to adopt a lot of the ideas or it wouldn’t have succeeded.
There are apparently many fractured subgroups who all nominally practice the same religion as well. But the only way to tell membership has to do with subtle cultural signals/body appearance. With neural implants people could mimic the preferred subgroup also...
I think it would keep hitting challenges. It reminds me of culturally the effect the pandemic had. The US culture based on in-person work, and everyone working or they are to be allowed to starve and become homeless, was suddenly at odds with reality.
Or just rules lawyering. Supposedly brothels in some Islamic countries have a priest on hand to temporarily marry all the couples. Alcohol may be outlawed but other pharmaceuticals mimic a similar effect. You could anticipate the end result is a modern liberal civilization that rules lawyers it’s way out of needing to adhere to the obsolete religious principles. (homosexuality? still banned but VR doesn’t count. burkhas? still required but it’s legal to project a simulated image of yourself on top...)
If they have a superhuman AGI, they can use it to predict all possible ways people might try to find workarounds for their commandments (e.g., gender transition, neural implants, etc.) and make them impossible.
If I understand you correctly, you are pointing to the fact that values/shards change in response to novel situations. Sure and perhaps even solar system-wide 1984-style regime would over time slowly morph into (something closer to) luxurious fully-automated gay space communism. But that’s a big maybe IMO. If we had good evidence that prosperity loosens and liberalizes societies across historical and cultures contexts plus perhaps solid models of axiological evolution (I don’t know if something like that even exists), my credence in that would be higher. Also, why not use AGI to fix your regime’s fundamental values or at least make them a stable attractor over thousands or millions of years.
(I feel like right now we are mostly doing adversarial worldbuilding)
First, this presupposes that for any amount of suffering there is some amount of pleasure/bliss/happiness/eudaimonia that could outweigh it. Not all LWers accept this, so it’s worth pointing that out.
But I don’t think the eternal paradise/mediocrity/hell scenario accurately represents what is likely to happen in that scenario. I’d be more worried about somebody using AGI to conquer the world and establish a stable totalitarian system built on some illiberal system, like shariah (according to Caplan, it’s totally plausible for global totalitarianism to persist indefinitely). If you get to post-scarcity, you may grant all your subjects UBI, all basic needs met, etc. (or you may not, if you decide that this policy contradicts Quran or hadith), but if your convictions are strong enough, women will still be forced to wear burkas, be basically slaves of their male kin etc. One could make an argument that abundance robustly promotes more liberal worldview, loosening of social norms, etc., but AFAIK there is no robust evidence for that.
This is meant just to illustrate that you don’t need an outgroup to impose a lot of suffering. Having a screwed up normative framework is just enough.
This family of scenarios is probably still better than AGI doom though.
Thanks for the post.
I kept thinking how a theocracy, assuming it did adopt all the advanced technology we are almost certain is possible but lack the ability to implement, could deal with all these challenges to it’s beliefs.
Half the population is being mistreated because they were born the wrong gender/the wrong ethnic subgroup? No problem, they’ll just go gender transition to the favored group. Total body replacement would be possible so there would be no way to tell who did this.
Sure the theocracy could ban the treatments but it conquered the solar system—it had to adopt a lot of the ideas or it wouldn’t have succeeded.
There are apparently many fractured subgroups who all nominally practice the same religion as well. But the only way to tell membership has to do with subtle cultural signals/body appearance. With neural implants people could mimic the preferred subgroup also...
I think it would keep hitting challenges. It reminds me of culturally the effect the pandemic had. The US culture based on in-person work, and everyone working or they are to be allowed to starve and become homeless, was suddenly at odds with reality.
Or just rules lawyering. Supposedly brothels in some Islamic countries have a priest on hand to temporarily marry all the couples. Alcohol may be outlawed but other pharmaceuticals mimic a similar effect. You could anticipate the end result is a modern liberal civilization that rules lawyers it’s way out of needing to adhere to the obsolete religious principles. (homosexuality? still banned but VR doesn’t count. burkhas? still required but it’s legal to project a simulated image of yourself on top...)
If they have a superhuman AGI, they can use it to predict all possible ways people might try to find workarounds for their commandments (e.g., gender transition, neural implants, etc.) and make them impossible.
If I understand you correctly, you are pointing to the fact that values/shards change in response to novel situations. Sure and perhaps even solar system-wide 1984-style regime would over time slowly morph into (something closer to) luxurious fully-automated gay space communism. But that’s a big maybe IMO. If we had good evidence that prosperity loosens and liberalizes societies across historical and cultures contexts plus perhaps solid models of axiological evolution (I don’t know if something like that even exists), my credence in that would be higher. Also, why not use AGI to fix your regime’s fundamental values or at least make them a stable attractor over thousands or millions of years.
(I feel like right now we are mostly doing adversarial worldbuilding)