An unworthy successor is a bad outcome which I feel is under-discussed.
I wonder if the talk about RSI which surpasses humanity precludes discussion of takeovers by spiky, locally effective but globally hopeless AI. If progress is continuous, and if AIs are better than humans at converting intelligence into power from speed and bulk, there will necessarily be a moment where we are smarter but less powerful. Most scenarios assume the moment will be brief, but maybe it will not be.
I don’t want superhuman AIs to leave humanity behind, but I strongly prefer that to a world of fine-tuned digital morlocks possessing the instruments to overthrow us, but without the general faculties that let us look up at the stars and hope for more.
I think the scenario “AI exterminates its creators, and then fails to colonize space (and either destroys itself clumsily or remains stuck on its planet of origin until it runs out of resources)” is a possible explanation for the lack or extraterrestrial intelligence.
An unworthy successor is a bad outcome which I feel is under-discussed.
I wonder if the talk about RSI which surpasses humanity precludes discussion of takeovers by spiky, locally effective but globally hopeless AI. If progress is continuous, and if AIs are better than humans at converting intelligence into power from speed and bulk, there will necessarily be a moment where we are smarter but less powerful. Most scenarios assume the moment will be brief, but maybe it will not be.
I don’t want superhuman AIs to leave humanity behind, but I strongly prefer that to a world of fine-tuned digital morlocks possessing the instruments to overthrow us, but without the general faculties that let us look up at the stars and hope for more.
I think the scenario “AI exterminates its creators, and then fails to colonize space (and either destroys itself clumsily or remains stuck on its planet of origin until it runs out of resources)” is a possible explanation for the lack or extraterrestrial intelligence.
I think this is the worst non-contrived tragedy I can imagine.