The crux is taking the idea of superintelligence seriously and looking at the implications. A lot of “techno-optimism” for example is about not taking this seriously, and thus not having a real problem to worry about. So literal extinction being attention-grabbing is not the reason to talk about it, same as it’s not about marketing for AI companies. It’s the model of what superintelligence actually is that is the reason.
In the long term (once there is actually a superintelligence, rather than the current harmless precursors to it), the scope of impact of AI is not bounded, what could affect a billion people soon affects the Earth as a whole (uncontacted tribes and all), and then the Sun, and the reachable universe.
That’s the main issue with addressing AGI risk as an engineering problem: a catastrophic failure has the kind of blast radius that doesn’t allow future iterative improvement. With any other technology, there will be opportunity to learn from mistakes and do better, again and again, until you succeed. But not here.
The crux is taking the idea of superintelligence seriously and looking at the implications. A lot of “techno-optimism” for example is about not taking this seriously, and thus not having a real problem to worry about. So literal extinction being attention-grabbing is not the reason to talk about it, same as it’s not about marketing for AI companies. It’s the model of what superintelligence actually is that is the reason.
In the long term (once there is actually a superintelligence, rather than the current harmless precursors to it), the scope of impact of AI is not bounded, what could affect a billion people soon affects the Earth as a whole (uncontacted tribes and all), and then the Sun, and the reachable universe.
That’s the main issue with addressing AGI risk as an engineering problem: a catastrophic failure has the kind of blast radius that doesn’t allow future iterative improvement. With any other technology, there will be opportunity to learn from mistakes and do better, again and again, until you succeed. But not here.