People currently understand the physical world sufficiently to see that supernatural claims are bogus, and so there is certainty about impossibility of developments predicated on supernatural. People know robust and general laws of physics that imply impossibility of perpetual motion, and so we can conclude in advance with great certainty that any perpetual motion engineering project is going to fail. Some long-standing problems in mathematics were attacked unsuccessfully for a long time, and so we know that making further progress on them is hard. In all these cases, there are specific pieces of positive knowledge that enable the inference of impossibility or futility of certain endeavors.
In contrast, a lot of questions concerning Friendly AI remain confusing and unexplored. It might turn out to be impossibly difficult to make progress on them, or else a simple matter of figuring out how to apply standard tools of mainstream mathematics. We don’t know, but neither do we have positive knowledge that implies impossibility or extreme difficulty of progress on these questions. In particular, the enormity of consequences does not imply extreme improbability of influencing those consequences. It looks plausible that the problem can be solved.
People currently understand the physical world sufficiently to see that supernatural claims are bogus, and so there is certainty about impossibility of developments predicated on supernatural. People know robust and general laws of physics that imply impossibility of perpetual motion, and so we can conclude in advance with great certainty that any perpetual motion engineering project is going to fail. Some long-standing problems in mathematics were attacked unsuccessfully for a long time, and so we know that making further progress on them is hard. In all these cases, there are specific pieces of positive knowledge that enable the inference of impossibility or futility of certain endeavors.
In contrast, a lot of questions concerning Friendly AI remain confusing and unexplored. It might turn out to be impossibly difficult to make progress on them, or else a simple matter of figuring out how to apply standard tools of mainstream mathematics. We don’t know, but neither do we have positive knowledge that implies impossibility or extreme difficulty of progress on these questions. In particular, the enormity of consequences does not imply extreme improbability of influencing those consequences. It looks plausible that the problem can be solved.