I think this is an interesting answer, and it does have some use even outside of the scenario, but I do think that the more likely answer to the problem probably rests upon the rareness of life, and in particular the eukaryote transition is probably the most likely great filter, because natural selection had to solve a coordination problem, combined with this step only happening once in earth’s history, compared to all the other hard steps.
That said, I will say some more on this topic, if only to share my models:
The universe might be too large for exponential growth to fill it up. It doesn’t seem plausible for self-replication to be faster than exponential in the long-run, and if the universe is sufficiently large (like, bigger than 101030 or so?) then it’s impossible—even with FTL—to kill everything, and again the scenario doesn’t work. I suppose an exception would be if there were some act that literally ends the entire universe immediately (thus killing everything without a need to replicate). Also, an extremely-large universe would require an implausibly-strong Great Filter for us to actually be the first this late.
The big out here is time travel, and in these scenarios, assuming logical inconsistencies are prevented by the time travel mechanism, there’s a non-trivial chance that trying to cause a logical inconsistency destroys the universe immediately:
AI Doom might not happen. If humanity is asserted to be not AI-doomed then this argument turns on its head and our existence (to at least the extent that we might not be the first) argues that either light-cone-breaking FTL is impossible or AI doom is a highly-unusual thing to happen to civilisations. This is sort of a weird point to mention since the whole scenario is an Outside View argument that AI Doom is likely, but how seriously to condition on these sorts of arguments is a matter of some dispute.
In general, I’m more confident in light-cone breaking FTL being impossible then AI doom is highly unusual, but conditional on light cone breaking FTL being possible, I’d assert that AI doom is quite unusual for civilizations (excluding institutional failures, because these cases don’t impact the absolute difficulty of the technical problem)
My big reason for this is I think instruction following is actually reasonably easy, and is enough to prevent existential risk on it’s own, and this doesn’t really require that much value alignment, for the purposes of existential risk.
In essence, I’m stating that value alignment isn’t very necessary in order for large civilizations with AI to come out that aren’t purely grabby, and while there is some value alignment necessary, it can be surprisingly small and bounded by a constant.
+1 for at least trying to do something, and also being surprisingly useful outside of the fermi paradox.
I think this is an interesting answer, and it does have some use even outside of the scenario, but I do think that the more likely answer to the problem probably rests upon the rareness of life, and in particular the eukaryote transition is probably the most likely great filter, because natural selection had to solve a coordination problem, combined with this step only happening once in earth’s history, compared to all the other hard steps.
That said, I will say some more on this topic, if only to share my models:
The big out here is time travel, and in these scenarios, assuming logical inconsistencies are prevented by the time travel mechanism, there’s a non-trivial chance that trying to cause a logical inconsistency destroys the universe immediately:
https://www.lesswrong.com/posts/EScmxJAHeJY5cjzAj/ssa-rejects-anthropic-shadow-too#Probability_pumping_and_time_travel
In general, I’m more confident in light-cone breaking FTL being impossible then AI doom is highly unusual, but conditional on light cone breaking FTL being possible, I’d assert that AI doom is quite unusual for civilizations (excluding institutional failures, because these cases don’t impact the absolute difficulty of the technical problem)
My big reason for this is I think instruction following is actually reasonably easy, and is enough to prevent existential risk on it’s own, and this doesn’t really require that much value alignment, for the purposes of existential risk.
In essence, I’m stating that value alignment isn’t very necessary in order for large civilizations with AI to come out that aren’t purely grabby, and while there is some value alignment necessary, it can be surprisingly small and bounded by a constant.
+1 for at least trying to do something, and also being surprisingly useful outside of the fermi paradox.