But yes, I think that one aspect was that initially people hoped that having a “provably friendly AI” would be close to being able to guarantee a good outcome, and the more they thought about it the more various caveats to that became clear, and I think this was a gradual process.
A possibility of the logic in question being contradictory is just one aspect which might invalidate the proof, and whether a formalization is sufficiently adequate is another such aspect, and whether one’s procedure of verifying the proof is adequate is yet another one, and whether having a proof can lull one into false sense of security is yet another very important aspect.
And when the stakes are existential, one really, really dislikes the idea of “pressing the go button” when the uncertainty is this high even in the presence of a proof.
I think this has been a gradual shift of thinking, and that’s one of the reasons why the thinking gradually became more pessimistic...
I see people are downvoting this particular comment.
Since this comment is making a few different points, please do criticize… It would be good to have more fine-grained feedback on this.
But yes, I think that one aspect was that initially people hoped that having a “provably friendly AI” would be close to being able to guarantee a good outcome, and the more they thought about it the more various caveats to that became clear, and I think this was a gradual process.
A possibility of the logic in question being contradictory is just one aspect which might invalidate the proof, and whether a formalization is sufficiently adequate is another such aspect, and whether one’s procedure of verifying the proof is adequate is yet another one, and whether having a proof can lull one into false sense of security is yet another very important aspect.
And when the stakes are existential, one really, really dislikes the idea of “pressing the go button” when the uncertainty is this high even in the presence of a proof.
I think this has been a gradual shift of thinking, and that’s one of the reasons why the thinking gradually became more pessimistic...