Some sort of stupid and callous outcome is likely to result. Maybe not specifically “self-modifying into a monster/zombie and trapping humanity in a dystopian prison”, but something in that reference class of outcomes.
Your and my beliefs/questions don’t feel like they’re even much coming into contact with each other… Like, you (and also other people) just keep repeating “something bad could happen”. And I’m like “yeah obviously something extremely bad could happen; maybe it’s even likely, IDK; and more likely, something very bad at the beginning of the reign would happen (Genghis spends is first 200 years doing more killing and raping); but what I’m ASKING is, what happens then?”.
If you’re saying
There is a VERY HIGH CHANCE that the emperor would PERMANENTLY put us into a near-zero value state or a negative-value state.
then, ok, you can say that, but I want to understand why; and I have some reasons (as presented) for thinking otherwise.
Your hypothesis is about the dynamics within human minds embedded in something like contemporary societies with lots of other diverse humans whom the rulers are forced to model for one reason or another.
My point is that evil, rash, or unwise decisions at the very start of the process are likely, and that those decisions are likely to irrevocably break the conditions in which the dynamics you hypothesize are possible. Make the minds in charge no longer human in the relevant sense, or remove the need to interact with/model other humans, etc.
In my view, it doesn’t strongly bear on the final outcome-distribution whether the “humans tend to become nicer to other humans over time” hypothesis is correct, because “the god-kings remain humans hanging around all the other humans in a close-knit society for millennia” is itself a very rare class of outcomes.
Your hypothesis is about the dynamics within human minds embedded in something like contemporary societies with lots of other diverse humans whom the rulers are forced to model for one reason or another.
Absolutely not, no. Humans want to be around (some) other people, so the emperor will choose to be so. Humans want to be [many core aspects of humanness, not necessarily per se, but individually], so the emperor will choose to be so. Yes, the emperor could want these insufficiently for my argument to apply, as I’ve said earlier. But I’m not immediately recalling anyone (you or others) making any argument that, with high or even substantial probability, the emperor would not want these things sufficiently for my question, about the long-run of these things, to be relevant.
Yes: some other people. The ideologically and morally aligned people, usually. Social/informational bubbles that screen away the rest of humanity, from which they only venture out if forced to (due to the need to earn money/control the populace, etc.). This problem seems to get worse as the ability to insulate yourself from other improves, as could be observed with modern internet-based informational bubbles or the surrounded-by-yes-men problem of dictators.
ASI would make this problem transcendental: there would truly be no need to ever bother with the people outside your bubble again, they could be wiped out or their management outsourced to AIs.
Past this point, you’re likely never returning to bothering about them. Why would you, if you can instead generate entire worlds of the kinds of people/entities/experiences you prefer? It seems incredibly unlikely that human social instincts can only be satisfied – or even can be best satisfied – by other humans.
It seems incredibly unlikely that human social instincts can only be satisfied – or even can be best satisfied – by other humans.
You’re 100% not understanding my argument, which is sorta fair because I didn’t lay it out clearly, but I think you should be doing better anyway.
Here’s a sketch:
Humans want to be human-ish and be around human-ish entities.
So the emperor will be human-ish and be around human-ish entities for a long time. (Ok, to be clear, I mean a lot of developmental / experiential time—the thing that’s relevant for thinking about how the emperor’s way of being trends over time.)
When being human-ish and around human-ish entities, core human shards continue to work.
When core human shards continue to work, MAYBE this implies EVENTUALLY adopting beneficence (or something else like cosmopolitanism), and hence good outcomes.
Since the emperor will be human-ish and be around human-ish entities for a long time, IF 4 obtains, then good outomes.
And then I give two IDEAS about 4 (communing->[universalist democracy], and [information increases]->understanding->caring).
I don’t know what’s making you think I don’t understand your argument. Also, I’ve never publicly stated that I’m opting into Crocker’s Rules, so while I happen not to particularly mind the rudeness, your general policy on that seems out of line here.
When being human-ish and around human-ish entities, core human shards continue to work
My argument is that the process you’re hypothesizing would be sensitive to the exact way of being human-ish, the exact classes of human-ish entities around, and the exact circumstances in which the emperor has to be around them.
As a plain and down-to-earth example, if a racist surrounds themselves with a hand-picked group of racist friends, do you expect them to eventually develop universal empathy, solely through interacting with said racist friends? Addressing your specific ideas: nobody in that group would ever need to commune with non-racists, nor have to bother learning more about non-racists. And empirically, such groups don’t seem to undergo spontaneous deradicalizations.
As a plain and down-to-earth example, if a racist surrounds themselves with a hand-picked group of racist friends, do you expect them to eventually develop universal empathy, solely through interacting with said racist friends? Addressing your specific ideas: nobody in that group would ever need to commune with non-racists, nor have to bother learning more about non-racists. And empirically, such groups don’t seem to undergo spontaneous deradicalizations.
As a plain and down-to-earth example, if a racist surrounds themselves with a hand-picked group of racist friends, do you expect them to eventually develop universal empathy, solely through interacting with said racist friends? Addressing your specific ideas: nobody in that group would ever need to commune with non-racists, nor have to bother learning more about non-racists. And empirically, such groups don’t seem to undergo spontaneous deradicalizations.
So what do you think happens when they are hanging out together, and they are in charge, and it has been 1,000 years or 1,000,000 years?
They keep each other radicalized forever as part of some transcendental social dynamic.
They become increasingly non-human as time goes on, small incremental modifications and personality changes building on each other, until they’re no longer human in the senses necessary for your hypothesis to apply.
I assume your counter-model involves them getting bored of each other and seeking diversity/new friends, or generating new worlds to explore/communicate with, with the generating processes not constrained to only generate racists, leading to the extremists interacting with non-extremists and eventually incrementally adopting non-extremist perspectives?
If yes, this doesn’t seem like the overdetermined way for things to go:
The generating processes would likely be skewed towards only generating things the extremists would find palatable, meaning more people sharing their perspectives/not seriously challenging whatever deeply seated prejudices they have. They’re there to have a good time, not have existential/moral crises.
They may make any number of modifications to themselves to make them no longer human-y in the relevant sense. Including by simply letting human-standard self-modification algorithms run for 10^3-10^6 years, becoming superhumanly radicalized.
They may address the “getting bored” part instead, periodically wiping their memories (including by standard human forgetting) or increasing each other’s capacity to generate diverse interactions.
Ok so they only generate racists and racially pure people. And they do their thing. But like, there’s no other races around, so the racism part sorta falls by the wayside. They’re still racially pure of course, but it’s usually hard to tell that they’re racist; sometimes they sit around and make jokes to feel superior over lesser races, but this is pretty hollow since they’re not really engaged in any type of race relations. Their world isn’t especially about all that, anymore. Now it’s about… what? I don’t know what to imagine here, but the only things I do know how to imagine involve unbounded structure (e.g. math, art, self-reflection, self-reprogramming). So, they’re doing that stuff. For a very long time. And the race thing just is not a part of their world anymore. Or is it? I don’t even know what to imagine there. Instead of having tastes about ethnicity, they develop tastes about questions in math, or literature. In other words, [the differences between people and groups that they care about] migrate from race to features of people that are involved in unbounded stuff. If the AGI has been keeping the racially impure in an enclosure all this time, at some point the racists might have a glance back, and say, wait, all the interesting stuff about people is also interesting about these people. Why not have them join us as well.
Past this point, you’re likely never returning to bothering about them. Why would you, if you can instead generate entire worlds of the kinds of people/entities/experiences you prefer? It seems incredibly unlikely that human social instincts can only be satisfied – or even can be best satisfied – by other humans.
For the same reason that most people (if given the power to do so) wouldn’t just replace their loved ones with their altered versions that are better along whatever dimensions the person judged them as deficient/imperfect.
Your and my beliefs/questions don’t feel like they’re even much coming into contact with each other… Like, you (and also other people) just keep repeating “something bad could happen”. And I’m like “yeah obviously something extremely bad could happen; maybe it’s even likely, IDK; and more likely, something very bad at the beginning of the reign would happen (Genghis spends is first 200 years doing more killing and raping); but what I’m ASKING is, what happens then?”.
If you’re saying
then, ok, you can say that, but I want to understand why; and I have some reasons (as presented) for thinking otherwise.
Your hypothesis is about the dynamics within human minds embedded in something like contemporary societies with lots of other diverse humans whom the rulers are forced to model for one reason or another.
My point is that evil, rash, or unwise decisions at the very start of the process are likely, and that those decisions are likely to irrevocably break the conditions in which the dynamics you hypothesize are possible. Make the minds in charge no longer human in the relevant sense, or remove the need to interact with/model other humans, etc.
In my view, it doesn’t strongly bear on the final outcome-distribution whether the “humans tend to become nicer to other humans over time” hypothesis is correct, because “the god-kings remain humans hanging around all the other humans in a close-knit society for millennia” is itself a very rare class of outcomes.
Absolutely not, no. Humans want to be around (some) other people, so the emperor will choose to be so. Humans want to be [many core aspects of humanness, not necessarily per se, but individually], so the emperor will choose to be so. Yes, the emperor could want these insufficiently for my argument to apply, as I’ve said earlier. But I’m not immediately recalling anyone (you or others) making any argument that, with high or even substantial probability, the emperor would not want these things sufficiently for my question, about the long-run of these things, to be relevant.
Yes: some other people. The ideologically and morally aligned people, usually. Social/informational bubbles that screen away the rest of humanity, from which they only venture out if forced to (due to the need to earn money/control the populace, etc.). This problem seems to get worse as the ability to insulate yourself from other improves, as could be observed with modern internet-based informational bubbles or the surrounded-by-yes-men problem of dictators.
ASI would make this problem transcendental: there would truly be no need to ever bother with the people outside your bubble again, they could be wiped out or their management outsourced to AIs.
Past this point, you’re likely never returning to bothering about them. Why would you, if you can instead generate entire worlds of the kinds of people/entities/experiences you prefer? It seems incredibly unlikely that human social instincts can only be satisfied – or even can be best satisfied – by other humans.
You’re 100% not understanding my argument, which is sorta fair because I didn’t lay it out clearly, but I think you should be doing better anyway.
Here’s a sketch:
Humans want to be human-ish and be around human-ish entities.
So the emperor will be human-ish and be around human-ish entities for a long time. (Ok, to be clear, I mean a lot of developmental / experiential time—the thing that’s relevant for thinking about how the emperor’s way of being trends over time.)
When being human-ish and around human-ish entities, core human shards continue to work.
When core human shards continue to work, MAYBE this implies EVENTUALLY adopting beneficence (or something else like cosmopolitanism), and hence good outcomes.
Since the emperor will be human-ish and be around human-ish entities for a long time, IF 4 obtains, then good outomes.
And then I give two IDEAS about 4 (communing->[universalist democracy], and [information increases]->understanding->caring).
I don’t know what’s making you think I don’t understand your argument. Also, I’ve never publicly stated that I’m opting into Crocker’s Rules, so while I happen not to particularly mind the rudeness, your general policy on that seems out of line here.
My argument is that the process you’re hypothesizing would be sensitive to the exact way of being human-ish, the exact classes of human-ish entities around, and the exact circumstances in which the emperor has to be around them.
As a plain and down-to-earth example, if a racist surrounds themselves with a hand-picked group of racist friends, do you expect them to eventually develop universal empathy, solely through interacting with said racist friends? Addressing your specific ideas: nobody in that group would ever need to commune with non-racists, nor have to bother learning more about non-racists. And empirically, such groups don’t seem to undergo spontaneous deradicalizations.
I expect they’d get bored with that.
So what do you think happens when they are hanging out together, and they are in charge, and it has been 1,000 years or 1,000,000 years?
One or both of:
They keep each other radicalized forever as part of some transcendental social dynamic.
They become increasingly non-human as time goes on, small incremental modifications and personality changes building on each other, until they’re no longer human in the senses necessary for your hypothesis to apply.
I assume your counter-model involves them getting bored of each other and seeking diversity/new friends, or generating new worlds to explore/communicate with, with the generating processes not constrained to only generate racists, leading to the extremists interacting with non-extremists and eventually incrementally adopting non-extremist perspectives?
If yes, this doesn’t seem like the overdetermined way for things to go:
The generating processes would likely be skewed towards only generating things the extremists would find palatable, meaning more people sharing their perspectives/not seriously challenging whatever deeply seated prejudices they have. They’re there to have a good time, not have existential/moral crises.
They may make any number of modifications to themselves to make them no longer human-y in the relevant sense. Including by simply letting human-standard self-modification algorithms run for 10^3-10^6 years, becoming superhumanly radicalized.
They may address the “getting bored” part instead, periodically wiping their memories (including by standard human forgetting) or increasing each other’s capacity to generate diverse interactions.
Ok so they only generate racists and racially pure people. And they do their thing. But like, there’s no other races around, so the racism part sorta falls by the wayside. They’re still racially pure of course, but it’s usually hard to tell that they’re racist; sometimes they sit around and make jokes to feel superior over lesser races, but this is pretty hollow since they’re not really engaged in any type of race relations. Their world isn’t especially about all that, anymore. Now it’s about… what? I don’t know what to imagine here, but the only things I do know how to imagine involve unbounded structure (e.g. math, art, self-reflection, self-reprogramming). So, they’re doing that stuff. For a very long time. And the race thing just is not a part of their world anymore. Or is it? I don’t even know what to imagine there. Instead of having tastes about ethnicity, they develop tastes about questions in math, or literature. In other words, [the differences between people and groups that they care about] migrate from race to features of people that are involved in unbounded stuff. If the AGI has been keeping the racially impure in an enclosure all this time, at some point the racists might have a glance back, and say, wait, all the interesting stuff about people is also interesting about these people. Why not have them join us as well.
For the same reason that most people (if given the power to do so) wouldn’t just replace their loved ones with their altered versions that are better along whatever dimensions the person judged them as deficient/imperfect.