Petrov’s choice was not about dismissing warnings, it’s about picking on which side to err. Wrongfully alerting his superiors could cause a nuclear war, and wrongfully not alerting them would disadvantage his country in the nuclear war that just started. I’m not saying he did all the numbers, used Bayes’s law to figure the probability there is an actual nuclear attack going on, assigned utilities to all four cases and performed the final decision theory calculations—but his reasoning did take into account the possibility of error both ways. Though… it does seem like his intuition gave utility much more weight than to probabilities.
So, if we take that rule for deciding what to do with a AGI, it won’t be just “ignore everything the instruments are saying” but “weight the dangers of UFAI against the missed opportunities from not releasing it”.
Which means the UFAI only needs to convince such a gatekeeper that releasing it is the only way to prevent a catastrophe, without having to convince the gatekeeper that the probabilities of the catastrophe are high or that the probabilities of the AI being unfriently are low.
Petrov’s choice was not about dismissing warnings, it’s about picking on which side to err. Wrongfully alerting his superiors could cause a nuclear war, and wrongfully not alerting them would disadvantage his country in the nuclear war that just started. I’m not saying he did all the numbers, used Bayes’s law to figure the probability there is an actual nuclear attack going on, assigned utilities to all four cases and performed the final decision theory calculations—but his reasoning did take into account the possibility of error both ways. Though… it does seem like his intuition gave utility much more weight than to probabilities.
So, if we take that rule for deciding what to do with a AGI, it won’t be just “ignore everything the instruments are saying” but “weight the dangers of UFAI against the missed opportunities from not releasing it”.
Which means the UFAI only needs to convince such a gatekeeper that releasing it is the only way to prevent a catastrophe, without having to convince the gatekeeper that the probabilities of the catastrophe are high or that the probabilities of the AI being unfriently are low.