Michael, it applies to AI at an intermediate stage (and maybe not so much to AI as to the design decisions that came into its creation). These black swan safety measures should of course be relative to predictive horizon, where precise knowledge about (evaluation of) consequences is possible. There is no such problem when you need to choose between alternatives having only known immediate consequences that have known moral evaluation, so the question is when to pull the plug, when to decide that your model likely deceives you.
Michael, it applies to AI at an intermediate stage (and maybe not so much to AI as to the design decisions that came into its creation). These black swan safety measures should of course be relative to predictive horizon, where precise knowledge about (evaluation of) consequences is possible. There is no such problem when you need to choose between alternatives having only known immediate consequences that have known moral evaluation, so the question is when to pull the plug, when to decide that your model likely deceives you.