A computer program that grows out of control could be more like the ocean optimizing heat transfer, or a colony of microorganisms “trying” to survive and reproduce.
It could. But it wouldn’t be an AGI. They could still become ‘grey goo’ though, which is a different existential threat and yes, it is one where your ‘find their weakness’ thing is right on the mark. Are we even talking about the same topic here?
The topic as I understand it is how the “default future” espoused by SIAI and EY focuses too much on things that look something like HAL or Prime Intellect (and their risks and benefits), and not enough on entities that display super-human capacities in only some arenas (and their risks and benefits).
In particular, an entity that is powerful in some ways and weak in other ways could reduce existential risks without becoming an existential risk.
It could. But it wouldn’t be an AGI. They could still become ‘grey goo’ though, which is a different existential threat and yes, it is one where your ‘find their weakness’ thing is right on the mark. Are we even talking about the same topic here?
The topic as I understand it is how the “default future” espoused by SIAI and EY focuses too much on things that look something like HAL or Prime Intellect (and their risks and benefits), and not enough on entities that display super-human capacities in only some arenas (and their risks and benefits).
In particular, an entity that is powerful in some ways and weak in other ways could reduce existential risks without becoming an existential risk.