I agree. There’s nothing magical about “once”. I almost wrote “once or twice”, but it didn’t sit well with the level of caution I would prefer be the norm. While your analysis seems correct, I am worried if that’s the plan.
I think a safety team shouldgo into things with the attitude that this type of thing is important a last-line-of-defense, but should never trigger. The plan should involve a strong argument that what’s being build is safe. In fact if this type of safeguard gets triggered, I would want the policy to be to go back to the drawing board, take the new information into account, and come up with a more well-argued plan. The new plan can have “never to be used” safeguards like this, but hopefully it has more and different ones this time.
If, on the other hand, a safety team goes in with the idea that John’s safeguard can be iterated a few times as you argue, then I anticipate them fooling themselves by iterating too many times and coming up with a plan that accidentally skirts the safeguard in some hard-to-notice way.
(I have no reason to expect these sorts of anticipations to be calibrated; I’m just thinking cautiously here.)
I agree. There’s nothing magical about “once”. I almost wrote “once or twice”, but it didn’t sit well with the level of caution I would prefer be the norm. While your analysis seems correct, I am worried if that’s the plan.
I think a safety team should go into things with the attitude that this type of thing is important a last-line-of-defense, but should never trigger. The plan should involve a strong argument that what’s being build is safe. In fact if this type of safeguard gets triggered, I would want the policy to be to go back to the drawing board, take the new information into account, and come up with a more well-argued plan. The new plan can have “never to be used” safeguards like this, but hopefully it has more and different ones this time.
If, on the other hand, a safety team goes in with the idea that John’s safeguard can be iterated a few times as you argue, then I anticipate them fooling themselves by iterating too many times and coming up with a plan that accidentally skirts the safeguard in some hard-to-notice way.
(I have no reason to expect these sorts of anticipations to be calibrated; I’m just thinking cautiously here.)
Yeah fully agreed.