One way to think about it is that progress in AI capabilities means ever bigger and nastier surprises. You find that your AIs can produce realistic but false prose in abundance, you find that they have an inner monologue capable of deciding whether to lie, you find that there are whole communities of people doing what their AIs tell them to do… And humanity has failed if this escalation results in a nasty surprise big enough that it’s fatal for human civilization, that happens before we get to a transhuman world that is nonetheless safe even for mere humans (e.g. Ilya Sutskever’s “plurality of humanity-loving AGIs”).
One way to think about it is that progress in AI capabilities means ever bigger and nastier surprises. You find that your AIs can produce realistic but false prose in abundance, you find that they have an inner monologue capable of deciding whether to lie, you find that there are whole communities of people doing what their AIs tell them to do… And humanity has failed if this escalation results in a nasty surprise big enough that it’s fatal for human civilization, that happens before we get to a transhuman world that is nonetheless safe even for mere humans (e.g. Ilya Sutskever’s “plurality of humanity-loving AGIs”).
Technology having unexpected side effects is an old story … which means it’s not killed us yet. The conclusion of certain doom still isn’t justified.