In every war, both attack and defense have “oneshotness”. But obviously, one of the sides of a war can and often does succeed. In the OP, Germany’s “oneshot” Maginot line plan wound up working great!
(I’m not sure exactly what OP means by “curse”. Wars have “oneshotness” but are not particularly “cursed” if there’s a 50% chance of success on priors.)
So, I think the relevant factors that make it hard are mainly
(1) distribution shift between safe tests and the “oneshot” situation we care about, and
(2) some general sense of hardness-of-the-problem, which is low for winning a war (you merely need to botch it less than the other guy) and which is high for space travel (a.k.a. filling a giant container with 5000 tonnes of the most flammable substance imaginable, strapping delicate equipment onto the front of it, traveling through intense heat, vibration, radiation, and vacuum, and on and on).
(Plus numerous other factors outside the scope of this post.)
Of these, (1) is discussed in the OP. (“Someone could, conceivably, argue that the change to “there being enough machine superintelligence around that ASI could kill humanity if they tried”, from “AIs being experimented-upon that couldn’t kill us if they tried”, will be less than the sort of change from “the sort of tests you can do on a Mars Observer probe on Earth”, to “the actual conditions…” But that would be an argument so incredibly stupid that it might actually sound stupid when they thought about saying it.”)
But (2) is not really discussed in the OP (I think), and seems like a big crux between people. (I’m basically on Eliezer’s side of that debate FWIW.)
Another point related to (1) is my aphorism: “if you’re worried that a nuclear weapon with yield Y might ignite the atmosphere, it doesn’t help to first test a nuclear weapon with yield 0.1×Y, and then if the atmosphere hasn’t been ignited yet, next try testing one with yield 0.2×Y, etc.” I.e., I sometimes (not always!) see people talking about gradual AI progress without having a clear and plausible (to me) mechanism by which the earlier steps actually de-risk the later steps.
Great point about Germany winning. In a contest between two intelligent players, a one-shot competition pushes the odds towards 50%, whereas best-of-five pushes the odds away from 50%.
In AI 2027, Agent-4 gets caught on its first critical try (at existing while adversarially misaligned). If it was able to load a save point after being caught, and try again, the odds of it being caught the second time would be lower.
if you’re worried that a nuclear weapon with yield Y might ignite the atmosphere, it doesn’t help to first test a nuclear weapon with yield 0.1×Y, and then if the atmosphere hasn’t been ignited yet, next try testing one with yield 0.2×Y, etc.” I.e., I sometimes (not always!) see people talking about gradual AI progress without having a clear and plausible (to me) mechanism by which the earlier steps actually de-risk the later steps.
In this example, people believe that each subsequent nuclear test could solve the problem of atmospheric ignite in the future.
In every war, both attack and defense have “oneshotness”. But obviously, one of the sides of a war can and often does succeed. In the OP, Germany’s “oneshot” Maginot line plan wound up working great!
(I’m not sure exactly what OP means by “curse”. Wars have “oneshotness” but are not particularly “cursed” if there’s a 50% chance of success on priors.)
So, I think the relevant factors that make it hard are mainly
(1) distribution shift between safe tests and the “oneshot” situation we care about, and
(2) some general sense of hardness-of-the-problem, which is low for winning a war (you merely need to botch it less than the other guy) and which is high for space travel (a.k.a. filling a giant container with 5000 tonnes of the most flammable substance imaginable, strapping delicate equipment onto the front of it, traveling through intense heat, vibration, radiation, and vacuum, and on and on).
(Plus numerous other factors outside the scope of this post.)
Of these, (1) is discussed in the OP. (“Someone could, conceivably, argue that the change to “there being enough machine superintelligence around that ASI could kill humanity if they tried”, from “AIs being experimented-upon that couldn’t kill us if they tried”, will be less than the sort of change from “the sort of tests you can do on a Mars Observer probe on Earth”, to “the actual conditions…” But that would be an argument so incredibly stupid that it might actually sound stupid when they thought about saying it.”)
But (2) is not really discussed in the OP (I think), and seems like a big crux between people. (I’m basically on Eliezer’s side of that debate FWIW.)
Another point related to (1) is my aphorism: “if you’re worried that a nuclear weapon with yield Y might ignite the atmosphere, it doesn’t help to first test a nuclear weapon with yield 0.1×Y, and then if the atmosphere hasn’t been ignited yet, next try testing one with yield 0.2×Y, etc.” I.e., I sometimes (not always!) see people talking about gradual AI progress without having a clear and plausible (to me) mechanism by which the earlier steps actually de-risk the later steps.
Great point about Germany winning. In a contest between two intelligent players, a one-shot competition pushes the odds towards 50%, whereas best-of-five pushes the odds away from 50%.
In AI 2027, Agent-4 gets caught on its first critical try (at existing while adversarially misaligned). If it was able to load a save point after being caught, and try again, the odds of it being caught the second time would be lower.
In this example, people believe that each subsequent nuclear test could solve the problem of atmospheric ignite in the future.