A take on values of the future, assuming AIs automate away politics and economics from humans.
There has been exploration on this topic before, like Jim Buhler’s What Values Will Control The Future Sequence and the appendix of relevant work here, as well as books like Foragers, Farmers and Fossil Fuels, which full disclosure, influenced a lot of my views on how values evolve, and gives a much more plausible picture than pictures that view value evolution as converging to CEV/moral truth, or that emphasize arbitrary societal factors for value evolution rather than energy considerations.
Indeed, I think it’s so plausible that we can actually non-trivially constrain post-AGI society values even without much empirical evidence.
However, given that AI that automates away humans is likely coming within at most the next 20-30 years, it’s worth thinking about what values will be dominant in the future least a bit.
While we still mostly don’t have very good predictions on what the post-AGI era will look like, we have uncovered some answers, and also have honed in on some important hinge questions, such that we aren’t completely blind to what values will be dominant in the future.
One of the central questions for a lot of value evolution boils down to “does acausal trade actually become practical for AIs to do in such a way that the constraints of previous governments requiring them to hold only territory that they can send armies to faster than rebellions can overthrow them?”
If the answer is yes, then AI values become more arbitrary and value lock-in, could in theory affect the entire accessible universe.
If the answer is no, then it’s a lot easier to constrain what the AI values, and value lock-in doesn’t matter, and alignment also matters less as a problem.
One particular example of this is that we can be pretty confident that people will probably be fine with ludicrously large amounts of inequality in both the political and economic dimensions, compared to any other societal type we had in history, and this includes even the farming era of human history, and the reason for this is that with advanced AI, the mechanisms that keep wealth and income inequalities in check will weaken to the point of no longer existing, and the ability for anyone else like developing countries to catch-up will end because their labor stops mattering, and natural resources can and are already owned by other rich actors in the world like corporations, which Phil Trammell talks about a lot more here.
The economic inequality could alone let us evolve into us valuing political inequality, via the wealthy buying up land to give themselves powers reserved to states, and them being able to defend their riches via robots, but one other issue is that while it probably isn’t a problem in the short to medium run, and is probably overrated as a problem from an alignment perspective, AIs that can genuinely persuade massive amounts of people to do stuff IRL/be superpersuasive is probably going to come in the longer-term, via 2 effects:
More citizens of states will be uploads by default, and uploaded brains are probably easier to hijack/jailbreak than current biological brains because you can reset them arbitrarily to a known and potentially even maximally vulnerable state, which isn’t possible to do for a biological brain so far (indeed a lot of jailbreaks/adversarial examples like KataGo adversarial attacks rely on the fact that it’s super easy to trick AIs/reset them continuously).
It’s probably going to be easier to modify citizens using all sorts of tools like genetics, nanotech, uploading and more, and this means rulers can erode the values of the population to the values their rulers want.
Also, AI will break the pattern of no one person ruling alone, at least assuming alignment is solved, because you can automate the police and militaries that would usually check your power away.
The level of inequality in economics and politics that many people will probably accept is closer to the inequalities between superheroes in modern comics vs the average citizen or mythic/non-Abrahamic religious gods ruling over a normal citizen class than basically any other society we’ve had in history, and the old deal described below will come back, but far, far more intensely and closer to the limiting process:
Especially revered was the “Old Deal”, Morris’ term for the generalised social contract between classes in agrarian societies: that some have the duty to be commanders (or “shepherds of the people”, in the preferred phrasing of many a king), others to obey those commands, and if everyone follows this script then things work fine.
Gender/sex inequality is an area where I expect the exact opposite trend to happen, and will continue the industrialist era trends, mostly because it will become more arbitrary and divorced from economic usefulness (indeed, in a fully automated away AI economy, gender roles do not matter anymore, and we don’t even need to reach the limiting process to have big impacts)
Attitudes to violence might be polarized, because on the large scale wars are inefficient and will get more inefficient relative to other outcomes like peaceful trade or defined borders which neither side will trespass, but on smaller scales war/murder/violence in general will have lower costs than ever before in all of history because of atomic precision manufacturing + backups making the average citizen way, way harder to kill (because now you have to destroy all backups rather than just ending their lives) compared to the benefits, which means the murder rate and the assault/serious violence/rape rate could end up diverging a lot.
That said, this isn’t as trivial to determine without empirical evidence, and thus I’m way less confident in this prediction than basically all of my other predictions to date.
If humans on the upper end of the economic/political inequality scale are there because of command over superhuman AI, what reason would they have to command the obedience (or protect the existence) of humans on the lower end of that scale? The kings of “the Old Deal” needed underlings to feed them and fight in their wars. Without that need, retaining underlings is a fetish, not a necessity — like hiring someone to wash dishes for you by hand instead of owning a dishwasher, or hiring a chauffeur instead of using a self-driving car.
I agree that strictly speaking, they don’t need to keep them alive anymore, and to be clear, this analysis holds almost as well if you replaced people with AI, with the exception of the points on violence, so most of the analysis doesn’t depend on people being around to live in it or being commanded.
This seems to inevitably lead to the conclusion that anyone who opposes genocide must oppose the creation of superhuman AI; or at least privately-controlled superhuman AI. (Which shouldn’t be a surprise from a classic AI-safety standpoint.)
I disagree with this conclusion, actually, because I didn’t say that AI developers or AIs themselves would attempt to exterminate humanity, I only said that my analysis was compatible with that outcome, and so was more general than you thought.
In order to reach this conclusion, you also need opinions on how likely this is to happen.
A take on values of the future, assuming AIs automate away politics and economics from humans.
There has been exploration on this topic before, like Jim Buhler’s What Values Will Control The Future Sequence and the appendix of relevant work here, as well as books like Foragers, Farmers and Fossil Fuels, which full disclosure, influenced a lot of my views on how values evolve, and gives a much more plausible picture than pictures that view value evolution as converging to CEV/moral truth, or that emphasize arbitrary societal factors for value evolution rather than energy considerations.
Indeed, I think it’s so plausible that we can actually non-trivially constrain post-AGI society values even without much empirical evidence.
However, given that AI that automates away humans is likely coming within at most the next 20-30 years, it’s worth thinking about what values will be dominant in the future least a bit.
While we still mostly don’t have very good predictions on what the post-AGI era will look like, we have uncovered some answers, and also have honed in on some important hinge questions, such that we aren’t completely blind to what values will be dominant in the future.
One of the central questions for a lot of value evolution boils down to “does acausal trade actually become practical for AIs to do in such a way that the constraints of previous governments requiring them to hold only territory that they can send armies to faster than rebellions can overthrow them?”
If the answer is yes, then AI values become more arbitrary and value lock-in, could in theory affect the entire accessible universe.
If the answer is no, then it’s a lot easier to constrain what the AI values, and value lock-in doesn’t matter, and alignment also matters less as a problem.
One particular example of this is that we can be pretty confident that people will probably be fine with ludicrously large amounts of inequality in both the political and economic dimensions, compared to any other societal type we had in history, and this includes even the farming era of human history, and the reason for this is that with advanced AI, the mechanisms that keep wealth and income inequalities in check will weaken to the point of no longer existing, and the ability for anyone else like developing countries to catch-up will end because their labor stops mattering, and natural resources can and are already owned by other rich actors in the world like corporations, which Phil Trammell talks about a lot more here.
The economic inequality could alone let us evolve into us valuing political inequality, via the wealthy buying up land to give themselves powers reserved to states, and them being able to defend their riches via robots, but one other issue is that while it probably isn’t a problem in the short to medium run, and is probably overrated as a problem from an alignment perspective, AIs that can genuinely persuade massive amounts of people to do stuff IRL/be superpersuasive is probably going to come in the longer-term, via 2 effects:
More citizens of states will be uploads by default, and uploaded brains are probably easier to hijack/jailbreak than current biological brains because you can reset them arbitrarily to a known and potentially even maximally vulnerable state, which isn’t possible to do for a biological brain so far (indeed a lot of jailbreaks/adversarial examples like KataGo adversarial attacks rely on the fact that it’s super easy to trick AIs/reset them continuously).
It’s probably going to be easier to modify citizens using all sorts of tools like genetics, nanotech, uploading and more, and this means rulers can erode the values of the population to the values their rulers want.
Also, AI will break the pattern of no one person ruling alone, at least assuming alignment is solved, because you can automate the police and militaries that would usually check your power away.
The level of inequality in economics and politics that many people will probably accept is closer to the inequalities between superheroes in modern comics vs the average citizen or mythic/non-Abrahamic religious gods ruling over a normal citizen class than basically any other society we’ve had in history, and the old deal described below will come back, but far, far more intensely and closer to the limiting process:
Gender/sex inequality is an area where I expect the exact opposite trend to happen, and will continue the industrialist era trends, mostly because it will become more arbitrary and divorced from economic usefulness (indeed, in a fully automated away AI economy, gender roles do not matter anymore, and we don’t even need to reach the limiting process to have big impacts)
Attitudes to violence might be polarized, because on the large scale wars are inefficient and will get more inefficient relative to other outcomes like peaceful trade or defined borders which neither side will trespass, but on smaller scales war/murder/violence in general will have lower costs than ever before in all of history because of atomic precision manufacturing + backups making the average citizen way, way harder to kill (because now you have to destroy all backups rather than just ending their lives) compared to the benefits, which means the murder rate and the assault/serious violence/rape rate could end up diverging a lot.
That said, this isn’t as trivial to determine without empirical evidence, and thus I’m way less confident in this prediction than basically all of my other predictions to date.
If humans on the upper end of the economic/political inequality scale are there because of command over superhuman AI, what reason would they have to command the obedience (or protect the existence) of humans on the lower end of that scale? The kings of “the Old Deal” needed underlings to feed them and fight in their wars. Without that need, retaining underlings is a fetish, not a necessity — like hiring someone to wash dishes for you by hand instead of owning a dishwasher, or hiring a chauffeur instead of using a self-driving car.
I agree that strictly speaking, they don’t need to keep them alive anymore, and to be clear, this analysis holds almost as well if you replaced people with AI, with the exception of the points on violence, so most of the analysis doesn’t depend on people being around to live in it or being commanded.
This seems to inevitably lead to the conclusion that anyone who opposes genocide must oppose the creation of superhuman AI; or at least privately-controlled superhuman AI. (Which shouldn’t be a surprise from a classic AI-safety standpoint.)
I disagree with this conclusion, actually, because I didn’t say that AI developers or AIs themselves would attempt to exterminate humanity, I only said that my analysis was compatible with that outcome, and so was more general than you thought.
In order to reach this conclusion, you also need opinions on how likely this is to happen.