I think exogneoue xrisk is probably very small, especially that I’m in favor of starting some chill levels od space colonization (e.g. Mars) pretty soon.
It’s not impossible but I would find it very surprising if your TED AI collective was unwise or untrustrworthy enough that it could drift into misalignment over time without stopping itself, but trustworthy and wise enough that we could have handed off the intelligrnt explosion to them, and they would have safely got us to the limits of intelligence.
Yep, the third point is a serious objection, but I feel that if our descendants never feel ready to buikd superintelligence, then maybe they just shouldn’t. I think it’s plausible that TED AI plus maybe narrow superintelligences (e.g. specialized in space-probe design) are enough for a near maximally good future.
I think exogneoue xrisk is probably very small, especially that I’m in favor of starting some chill levels od space colonization (e.g. Mars) pretty soon.
It’s not impossible but I would find it very surprising if your TED AI collective was unwise or untrustrworthy enough that it could drift into misalignment over time without stopping itself, but trustworthy and wise enough that we could have handed off the intelligrnt explosion to them, and they would have safely got us to the limits of intelligence.
Yep, the third point is a serious objection, but I feel that if our descendants never feel ready to buikd superintelligence, then maybe they just shouldn’t. I think it’s plausible that TED AI plus maybe narrow superintelligences (e.g. specialized in space-probe design) are enough for a near maximally good future.