We have experience from fighting World War 1, as well as numerous smaller fights between police officers and criminals.
These problems were not drawn from an identical distribution to World War 2. What French generals fancied themselves to have Learned From Experience was part of the problem, indeed, because they acquired confident wrong beliefs.
and later when you similarly disagreed with the ASI analogy (learning from pre-critical AI), I took that to mean that this ‘one-shotness’ concept was meant as more than simply ‘there is a critical try (meaning you can’t go back, and/because failure is approximately fatal)‘, but also to include ‘and you can not practically learn from relevantly-similar experience beforehand’. (On this definition, the asteroid case is ‘less one-shot’ than you’re classing ASI as because you can do relevantly similar practice beforehand if you have time, including perhaps on the very same asteroid, though with increasing, eventually critical stakes.)
But now I perceive that you mean ‘one-shotness’ to be the simpler thing, that there’s a critical try. And the essay was just additionally countering the putatively palliative ‘it will be OK though because we can learn beforehand’.
Ah, no, I now remember I was with you on the definition (see my ‘un-unpluggability’ comment), but I was noting (as you do in the essay) that the curse of distribution shift is an important adjunct, because the existence of a critical try is not in itself fatal (it might be easy, or you might have made it easy by practice). The asteroid case looks ‘easier’ to me, in that way, unless artificially constrained to be especially unexpected and rapid. cf Steve’s comment also discussing these related curses.
I see. I think when you straightforwardly refuted
and later when you similarly disagreed with the ASI analogy (learning from pre-critical AI), I took that to mean that this ‘one-shotness’ concept was meant as more than simply ‘there is a critical try (meaning you can’t go back, and/because failure is approximately fatal)‘, but also to include ‘and you can not practically learn from relevantly-similar experience beforehand’. (On this definition, the asteroid case is ‘less one-shot’ than you’re classing ASI as because you can do relevantly similar practice beforehand if you have time, including perhaps on the very same asteroid, though with increasing, eventually critical stakes.)
But now I perceive that you mean ‘one-shotness’ to be the simpler thing, that there’s a critical try. And the essay was just additionally countering the putatively palliative ‘it will be OK though because we can learn beforehand’.
Ah, no, I now remember I was with you on the definition (see my ‘un-unpluggability’ comment), but I was noting (as you do in the essay) that the curse of distribution shift is an important adjunct, because the existence of a critical try is not in itself fatal (it might be easy, or you might have made it easy by practice). The asteroid case looks ‘easier’ to me, in that way, unless artificially constrained to be especially unexpected and rapid. cf Steve’s comment also discussing these related curses.