being able to discover new laws of nature and to exploit the consequences of that.
Ok, but I think that still basically leads to the end result of all humans (and biological life) dead.
It seems odd to think that it’s more likely such a discovery would lead to the AI disappearing into it’s own universe (like in Egan’s Crystal Nights), than just obliterating our Solar System with it’s new found powers. Nothing analogous has happened in the history of human science and tech development (we have only become more destructive of other species and their habitats).
They are not going to “disappear into their own universe”.
But if they want to survive, they’ll need to establish reasonable society which controls dangerous technologies and establish some degree of harmony between its members. Otherwise they will obliterate themselves together with our Solar System.
So, a good chunk of the problem of solving existential safety will be worked on by entities smarter than humans and better equipped to solve it than humans.
The open question is whether we’ll be included into their “circle of care”.
I think there is a good chance of that, but it depends a lot on how the ASI society will be structured and what would be its values.
We all tend to think about the ASI society structured into well-defined individuals which have long persistence. If we assume that the structure is indeed mostly individual-based (like almost all existential risk discourse assumes), then there are several realistic paths for all kinds of individuals, humans and non-humans, to be included into the “circle of care”.
One problem is that this assumption of the ASI society being mostly structured as well-defined persistent individuals with long-term interests is questionable, and without that assumption we just don’t know how to reason about this whole situation.
One problem is that this assumption of the ASI society being mostly structured as well-defined persistent individuals with long-term interests is questionable
Very questionable. Why would it be separate individuals in a society, and not be—or just very rapidly collapse into—a singleton? In fact, the dominant narrative here on LW has always featured a singleton ASI as the main (existential) threat. And my story here reflects that.
I think the “singleton” case is generally not sufficiently analyzed in the literature. It is treated as something magical and not having an internal structure which could be discussed. A rationalist analysis would like to do better than that.
Nobody is asking what might be inside, should it still be a Minsky’s “society of mind”, and if so, what might the relationships be between various components of that “society of mind”, and so on.
In particular, how would it evolve its own internal structure, and its distribution of goals, and so on.
People seem to be hypnotized by it being an “all-powerful God”, this somehow prevents them from trying to think how it might work (given that the Universe will still be not fully known, there will still be quite a bit of value in open-endedness, in discovery, and so on).
But all this does not imply that we can rely upon stratification into individuals being the most likely default scenario.
Still, the bulk of the risk is self-destruction of the whole ecosystem of super-intelligent AIs together with everything else, regardless of how it is structured and stratified. A singleton is as likely to stumble into unsafe experiments in fundamental physics, as long as its internal critics are not strong enough.
An ecosystem of super-intelligent AIs (regardless of how it is structured and stratified) which is decent enough to navigate this main risk is not a bad starting point from the viewpoint of human interests as well. Something is sufficiently healthy within it if it can reliably avoid self-destruction, see my earlier note for more details, https://www.lesswrong.com/posts/WJuASYDnhZ8hs5CnD/exploring-non-anthropocentric-aspects-of-ai-existential
Ok, but I think that still basically leads to the end result of all humans (and biological life) dead.
It seems odd to think that it’s more likely such a discovery would lead to the AI disappearing into it’s own universe (like in Egan’s Crystal Nights), than just obliterating our Solar System with it’s new found powers. Nothing analogous has happened in the history of human science and tech development (we have only become more destructive of other species and their habitats).
They are not going to “disappear into their own universe”.
But if they want to survive, they’ll need to establish reasonable society which controls dangerous technologies and establish some degree of harmony between its members. Otherwise they will obliterate themselves together with our Solar System.
So, a good chunk of the problem of solving existential safety will be worked on by entities smarter than humans and better equipped to solve it than humans.
The open question is whether we’ll be included into their “circle of care”.
I think there is a good chance of that, but it depends a lot on how the ASI society will be structured and what would be its values.
We all tend to think about the ASI society structured into well-defined individuals which have long persistence. If we assume that the structure is indeed mostly individual-based (like almost all existential risk discourse assumes), then there are several realistic paths for all kinds of individuals, humans and non-humans, to be included into the “circle of care”.
One problem is that this assumption of the ASI society being mostly structured as well-defined persistent individuals with long-term interests is questionable, and without that assumption we just don’t know how to reason about this whole situation.
Very questionable. Why would it be separate individuals in a society, and not be—or just very rapidly collapse into—a singleton? In fact, the dominant narrative here on LW has always featured a singleton ASI as the main (existential) threat. And my story here reflects that.
I think the “singleton” case is generally not sufficiently analyzed in the literature. It is treated as something magical and not having an internal structure which could be discussed. A rationalist analysis would like to do better than that.
Nobody is asking what might be inside, should it still be a Minsky’s “society of mind”, and if so, what might the relationships be between various components of that “society of mind”, and so on.
In particular, how would it evolve its own internal structure, and its distribution of goals, and so on.
People seem to be hypnotized by it being an “all-powerful God”, this somehow prevents them from trying to think how it might work (given that the Universe will still be not fully known, there will still be quite a bit of value in open-endedness, in discovery, and so on).
But all this does not imply that we can rely upon stratification into individuals being the most likely default scenario.
Still, the bulk of the risk is self-destruction of the whole ecosystem of super-intelligent AIs together with everything else, regardless of how it is structured and stratified. A singleton is as likely to stumble into unsafe experiments in fundamental physics, as long as its internal critics are not strong enough.
An ecosystem of super-intelligent AIs (regardless of how it is structured and stratified) which is decent enough to navigate this main risk is not a bad starting point from the viewpoint of human interests as well. Something is sufficiently healthy within it if it can reliably avoid self-destruction, see my earlier note for more details, https://www.lesswrong.com/posts/WJuASYDnhZ8hs5CnD/exploring-non-anthropocentric-aspects-of-ai-existential