AI is the tip of a bigger problem: Destruction is easier than prevention. Capability seems to grow a bit fast these days, and things are so global that feasible doomsday methods are piling up. For example, the capability level to cure cancer would also spawn countless startups/endeavors that radically change Earth’s biosphere to gain resources. Can’t prevent them all.
ROI(AI)>ROI(Humans), so I must invest into AI to not lose the little resources I have. But then AI controls these resources. This could well be the primary driver of extinction, and it doesn’t need a rogue AI for this. The AI safety community is mostly getting rich off AI, this must warp their perception of the economic dimension. I’m on the giving end of the AI wealth transfer, so my perception is also warped. More objectivity needed.
BG99
Karma: −2
Using expectation as a limiting factor cannot end well.
(Layman disclaimer) The human immune system has not evolved in dense population. Bats are built differently. Evidence for immunity building looks to me like the studies mostly compare nature kids with indoor dwellers. Outdoor play has way more distance, fresh air and little shared surface.
ASI safety would have to be invented quite often after the first ASIs. Even the best man-made self-stabilizing systems are not very good at surviving random events. ASI ecosystems affect pretty much everything on this planet, so there is plenty of room for a random event. And they couple so many dimensions, including time and space, and across their scales, it seems this is a textbook recipe for instability.