Granted, it’s a great plot device once you give it superpowers, and so there have been many high profile movies concerning such scenarios, and you see worlds destroyed by AI on the big screen. And your internal probability evaluator—evolved before CGI—uses the frequency of scenarios you seen with your own eyes.
PM’s precise point is also raised in the Sequences.
Yes, and John’s point is that just because it’s possible to arrive at that conclusion by a wrong method doesn’t actually mean it’s wrong.