Which means that if we buy this [great filter derivation] argument, we should put a lot more weight on the category of ‘everything else’, and especially the bits of it that come before AI. To the extent that known risks like biotechnology and ecological destruction don’t seem plausible, we should more fear unknown unknowns that we aren’t even preparing for.
True in principle. I do think that the known risks don’t cut it; some of them might be fairly deadly, but even in aggregate they don’t look nearly deadly enough to contribute much to the great filter. Given the uncertainties in the great filter analysis, that conclusion for me mostly feeds back in that direction, increasing the probability that the GF is in fact behind us.
Your SIA doomsday argument—as pointed out by michael vassar in the comments—has interesting interactions with the simulation hypothesis; specifically, since we don’t know if we’re in a simulation, the bayesian update in step 3 can’t be performed as confidently as you stated. Given this, “we really can’t see a plausible great filter coming up early enough to prevent us from hitting superintelligence” is also evidence for this environment being a simulation.
True in principle. I do think that the known risks don’t cut it; some of them might be fairly deadly, but even in aggregate they don’t look nearly deadly enough to contribute much to the great filter. Given the uncertainties in the great filter analysis, that conclusion for me mostly feeds back in that direction, increasing the probability that the GF is in fact behind us.
Your SIA doomsday argument—as pointed out by michael vassar in the comments—has interesting interactions with the simulation hypothesis; specifically, since we don’t know if we’re in a simulation, the bayesian update in step 3 can’t be performed as confidently as you stated. Given this, “we really can’t see a plausible great filter coming up early enough to prevent us from hitting superintelligence” is also evidence for this environment being a simulation.