Since a powerful AI would likely spread its influence through its future lightcone, rogue AI are not likely to be a major part of the Great Filter (although Doomsday Argument style anthropic reasoning/ observer considerations do potentially imply problems in the future of which could include AI). One major suggested existential risk/filtration issue is nanotech. Moreover, easy nanotech is a major part of many scenarios of AIs going foom. Given this, should the SIAI be evaluating the practical limitations and risks of nanotech, or are there enough groups already doing so?
The first point looks like this one. The case for the Doomsday Argument implying problems looks weak to me. It just says that there (probably) won’t be lots of humans around in the future. However, IMO, that is pretty obvious—humans are unlikely to persist far into an engineered future.
Since a powerful AI would likely spread its influence through its future lightcone, rogue AI are not likely to be a major part of the Great Filter (although Doomsday Argument style anthropic reasoning/ observer considerations do potentially imply problems in the future of which could include AI). One major suggested existential risk/filtration issue is nanotech. Moreover, easy nanotech is a major part of many scenarios of AIs going foom. Given this, should the SIAI be evaluating the practical limitations and risks of nanotech, or are there enough groups already doing so?
The first point looks like this one. The case for the Doomsday Argument implying problems looks weak to me. It just says that there (probably) won’t be lots of humans around in the future. However, IMO, that is pretty obvious—humans are unlikely to persist far into an engineered future.