A third-party guess: It’s coming from a flaw in the formal specification of the epistemic procedure. That it is formally specified is not a guarantee that it is the specification we would want. It could rest on a faulty assumption, or take a step that appears justified but in actuality is slightly wrong.
Basically, formal specification is a good idea, but not a get-out-of-trouble-free card.
Replying elsewhere. Suffice to say, nobody would call it a “get out of trouble free” card. More like, get out of trouble after decades of prerequisite hard work, which is precisely why various forms of the hard work are being done now, decades before any kind of AGI is invented, let alone foom-flavored ultra-AI.
A third-party guess: It’s coming from a flaw in the formal specification of the epistemic procedure. That it is formally specified is not a guarantee that it is the specification we would want. It could rest on a faulty assumption, or take a step that appears justified but in actuality is slightly wrong.
Basically, formal specification is a good idea, but not a get-out-of-trouble-free card.
Replying elsewhere. Suffice to say, nobody would call it a “get out of trouble free” card. More like, get out of trouble after decades of prerequisite hard work, which is precisely why various forms of the hard work are being done now, decades before any kind of AGI is invented, let alone foom-flavored ultra-AI.
Reply.
I have no idea if this is the communication failure, but I certainly would agree with this comment.
Thanks!