You use the Copernican principle (along with the fact that there are almost certainly billions of planetary systems in our past light cone) to conclude that (1) it is unlikely that we’re the only technological civilization in our past light cone. Then you go on to use the Fermi paradox. But why in your mind does the Fermi paradox not lead you to believe that we probably are the only civilization in our past light cone (in spite of the Copernican principle)?
In other words, aren’t you cherry picking by letting your argument rely on the lack of any evidence of a civilization-destroying AI’s having reached our solar system while acting as if your argument (and in particular the component of your argument I have labelled (1) above) is immune to the lack of any evidence of a (non-human) non-destroyed civilization’s having reached our solar system? Do you claim that an AI that destroyed the civilization that created it (if such existed in our past light cone) would have been more likely to expand than a civilization that avoided being destroyed would have been?
And I don’t yet see how the concept of a Great Filter throws any light on the question you are wrestling with. If there is a Great Filter that prevents almost every simple form of life from evolving into a galaxy-spanning civilization, then why not just conclude that humanity has already passed this Great Filter?
I suspect that the OP’s argument implies that we should double down on priors which imply the Great Filter and are yet consistent with observations. Suppose that the Milky way has habitable planets and that any civilisation generates observers before either destroying itself with probability 50% or reaching the maximal technological level. Also assume that intergalactic travel is impossible and that we have a probability to be in a universe where interstellar travel is impossible, while all other universes cause the first civilisation which reached the maximal tech level to prevent the emergence of others and to generate observers. Then the universe with no interstellar travel would generate observers on a stage like ours, while the universe with interstellar travel would generate such observers, meaning that a random observer like us is 50 times more likely to be in a universe without interstellar travel.
On the other hand, if the probability that interstellar travel is impossible was then an observer like us would be times MORE likely to be in a universe WITH interstellar travel, and changing the number of habitable reachable planets to (which is around the number of such planets in the entire reachable universe!) would mean that we are times more likely to be in a universe with interstellar travel.
What I struggle to understand is how many bits of evidence already point a perfect Bayesian in the direction of interstellar travel being possible. In a manner similar to Yudkowsky, I suspect that the amount of evidence pointing towards known physics is at least on the order of thousands of bits.
Well, we do know that interstellar travel is possible, just currently very hard. It’s a technological issue, not a physical one. I can certainly imagine slow travel, a few thousands km/s, being possible with nuclear rocket propulsion or other tech. Getting a spaceship-size object traveling at an appreciable fraction of the speed of light is a different story, anything like that would create observational artifacts visible from far away, so those probably aren’t anywhere in the observed past light cone. But I am not sure how this is related to the evaluation of the odds of AIpocalypse.
All good questions! We might be the only recognizably technological civilization out there, any other “life” might occupy something we cannot even tell is life.
I guess my point is that, provided we are not the only ones like us to begin with, concluding that we are still the unique ones that passed the Great Filter makes us pretty unique, and the question becomes “why?”. If AI x-risk is the great filter, then we would see the artifacts of it out there, unless the AGI magically stops growing somewhere between the current level and the observationally visible one.
A clever arguer wanting to persuade you that AI is dangerous can construct an argument much like yours: among the at least 10^22 stars in our past light cone, a decent number probably birthed a technological civilization. And maybe some of those civilizations had no interest in expanding. But surely some wanted to expand and to keep on expanding till they reached our solar system—if they could have. The most like explanation for why an alien civilization in our past light cone failed in their desire to expand and to keep on expanding is that their whole civilization went extinct. There is something dangerous about the universe that destroys civilizations. We don’t know for sure what that might be, but one strong possibility is that messing with AI is the primary destroyer of civilizations, so we should be more hesitant to mess with AI than we would have been if the skies were opaque and we had no idea other star system existed.
I’m not saying that that is a sound argument. It is not because it ignores the fact that after destroying the civilization that created it, an AI will tend to want to expand into space, too, just like a civilization would, and we see no signs of an AI’s having reach our star system. But your argument is equally unsound because it ignores the fact that we see no signs of any AI-surviving civilization’s having reached our star system.
The correct conclusion to draw from our observations of the cosmos (and from the lack of any signs of an alien visit to our planet) is that we were mistaken in assuming that other star systems are similar to our star system. It is fine to assume that those 10^22 stars (which are definitely real, no doubt about that) are like our star until we encounter enough evidence that they are in fact not like our star. And we have that evidence, e.g., all the evidence we have that allows us to get very concrete about the nature of interstellar travel and of a interstellar probe or space ship’s arrival at a new suitable star system followed by exploitation of the new system’s resources to build more probes.
I can imagine that an AI first destroys humanity, then poofs away before reaching for the stars, for sure. My point is that the window is quite narrow there.
I can also appreciate the argument that we are special and a space-faring civilization, artificial or otherwise, did not arise until now, or at least not in any visible to us ways, let alone a lightcone-consuming way. But that argument is not really new.
You use the Copernican principle (along with the fact that there are almost certainly billions of planetary systems in our past light cone) to conclude that (1) it is unlikely that we’re the only technological civilization in our past light cone. Then you go on to use the Fermi paradox. But why in your mind does the Fermi paradox not lead you to believe that we probably are the only civilization in our past light cone (in spite of the Copernican principle)?
In other words, aren’t you cherry picking by letting your argument rely on the lack of any evidence of a civilization-destroying AI’s having reached our solar system while acting as if your argument (and in particular the component of your argument I have labelled (1) above) is immune to the lack of any evidence of a (non-human) non-destroyed civilization’s having reached our solar system? Do you claim that an AI that destroyed the civilization that created it (if such existed in our past light cone) would have been more likely to expand than a civilization that avoided being destroyed would have been?
And I don’t yet see how the concept of a Great Filter throws any light on the question you are wrestling with. If there is a Great Filter that prevents almost every simple form of life from evolving into a galaxy-spanning civilization, then why not just conclude that humanity has already passed this Great Filter?
I suspect that the OP’s argument implies that we should double down on priors which imply the Great Filter and are yet consistent with observations. Suppose that the Milky way has
On the other hand, if the probability that interstellar travel is impossible was
What I struggle to understand is how many bits of evidence already point a perfect Bayesian in the direction of interstellar travel being possible. In a manner similar to Yudkowsky, I suspect that the amount of evidence pointing towards known physics is at least on the order of thousands of bits.
Well, we do know that interstellar travel is possible, just currently very hard. It’s a technological issue, not a physical one. I can certainly imagine slow travel, a few thousands km/s, being possible with nuclear rocket propulsion or other tech. Getting a spaceship-size object traveling at an appreciable fraction of the speed of light is a different story, anything like that would create observational artifacts visible from far away, so those probably aren’t anywhere in the observed past light cone. But I am not sure how this is related to the evaluation of the odds of AIpocalypse.
All good questions! We might be the only recognizably technological civilization out there, any other “life” might occupy something we cannot even tell is life.
I guess my point is that, provided we are not the only ones like us to begin with, concluding that we are still the unique ones that passed the Great Filter makes us pretty unique, and the question becomes “why?”. If AI x-risk is the great filter, then we would see the artifacts of it out there, unless the AGI magically stops growing somewhere between the current level and the observationally visible one.
A clever arguer wanting to persuade you that AI is dangerous can construct an argument much like yours: among the at least 10^22 stars in our past light cone, a decent number probably birthed a technological civilization. And maybe some of those civilizations had no interest in expanding. But surely some wanted to expand and to keep on expanding till they reached our solar system—if they could have. The most like explanation for why an alien civilization in our past light cone failed in their desire to expand and to keep on expanding is that their whole civilization went extinct. There is something dangerous about the universe that destroys civilizations. We don’t know for sure what that might be, but one strong possibility is that messing with AI is the primary destroyer of civilizations, so we should be more hesitant to mess with AI than we would have been if the skies were opaque and we had no idea other star system existed.
I’m not saying that that is a sound argument. It is not because it ignores the fact that after destroying the civilization that created it, an AI will tend to want to expand into space, too, just like a civilization would, and we see no signs of an AI’s having reach our star system. But your argument is equally unsound because it ignores the fact that we see no signs of any AI-surviving civilization’s having reached our star system.
The correct conclusion to draw from our observations of the cosmos (and from the lack of any signs of an alien visit to our planet) is that we were mistaken in assuming that other star systems are similar to our star system. It is fine to assume that those 10^22 stars (which are definitely real, no doubt about that) are like our star until we encounter enough evidence that they are in fact not like our star. And we have that evidence, e.g., all the evidence we have that allows us to get very concrete about the nature of interstellar travel and of a interstellar probe or space ship’s arrival at a new suitable star system followed by exploitation of the new system’s resources to build more probes.
I can imagine that an AI first destroys humanity, then poofs away before reaching for the stars, for sure. My point is that the window is quite narrow there.
I can also appreciate the argument that we are special and a space-faring civilization, artificial or otherwise, did not arise until now, or at least not in any visible to us ways, let alone a lightcone-consuming way. But that argument is not really new.
I might be missing your point...
Bluntly stated, my point is that the proper amount of weight to give to your argument for “why I am not too worried about AIpocalypse” is zero.