The asteroid case could be considered multi-shottable, if we had enough advance warning and space tech and went around practising asteroid-deflection long enough in advance. (I realise Matthew’s case posits ‘very soon’.) I think we’d in principle be able to get enough, generalisable-enough insight into asteroid deflection. Of course there’s a first ‘critical’ try (and we’d want not to deflect asteroids into Earth on the practice spree!). It’s just ‘deflecting mostly-ballistic space rocks’, which surely generalises well.
I think you’re distinguishing that sort of case from ASI because you consider any pre-critical evidence we can gather to be almost inevitably sufficiently out of distribution that it’s worth very little. Right? In particular, unlike the asteroid case, you might say that even with heaps of advance warning, there isn’t a test environment that’s sufficiently realistic, and there’s no realistic isolation region for ASI (unlike, say, ‘messing with asteroids far from Earth’)?
“I think you’re distinguishing War from the ongoing struggle between police and criminals because you think that in War any pre-critical evidence we can gather to be almost inevitably sufficiently out of distribution that it’s worth very little.”
No! The thing that makes the Maginot Line different from police enforcement in a random city is that if the Maginot Line fails the country falls and you don’t get to try again; not that War is changing much faster than criminal operations. War changes fast enough.
We have experience from fighting World War 1, as well as numerous smaller fights between police officers and criminals.
These problems were not drawn from an identical distribution to World War 2. What French generals fancied themselves to have Learned From Experience was part of the problem, indeed, because they acquired confident wrong beliefs.
and later when you similarly disagreed with the ASI analogy (learning from pre-critical AI), I took that to mean that this ‘one-shotness’ concept was meant as more than simply ‘there is a critical try (meaning you can’t go back, and/because failure is approximately fatal)‘, but also to include ‘and you can not practically learn from relevantly-similar experience beforehand’. (On this definition, the asteroid case is ‘less one-shot’ than you’re classing ASI as because you can do relevantly similar practice beforehand if you have time, including perhaps on the very same asteroid, though with increasing, eventually critical stakes.)
But now I perceive that you mean ‘one-shotness’ to be the simpler thing, that there’s a critical try. And the essay was just additionally countering the putatively palliative ‘it will be OK though because we can learn beforehand’.
Ah, no, I now remember I was with you on the definition (see my ‘un-unpluggability’ comment), but I was noting (as you do in the essay) that the curse of distribution shift is an important adjunct, because the existence of a critical try is not in itself fatal (it might be easy, or you might have made it easy by practice). The asteroid case looks ‘easier’ to me, in that way, unless artificially constrained to be especially unexpected and rapid. cf Steve’s comment also discussing these related curses.
The asteroid case could be considered multi-shottable, if we had enough advance warning and space tech and went around practising asteroid-deflection long enough in advance. (I realise Matthew’s case posits ‘very soon’.) I think we’d in principle be able to get enough, generalisable-enough insight into asteroid deflection. Of course there’s a first ‘critical’ try (and we’d want not to deflect asteroids into Earth on the practice spree!). It’s just ‘deflecting mostly-ballistic space rocks’, which surely generalises well.
I think you’re distinguishing that sort of case from ASI because you consider any pre-critical evidence we can gather to be almost inevitably sufficiently out of distribution that it’s worth very little. Right? In particular, unlike the asteroid case, you might say that even with heaps of advance warning, there isn’t a test environment that’s sufficiently realistic, and there’s no realistic isolation region for ASI (unlike, say, ‘messing with asteroids far from Earth’)?
“I think you’re distinguishing War from the ongoing struggle between police and criminals because you think that in War any pre-critical evidence we can gather to be almost inevitably sufficiently out of distribution that it’s worth very little.”
No! The thing that makes the Maginot Line different from police enforcement in a random city is that if the Maginot Line fails the country falls and you don’t get to try again; not that War is changing much faster than criminal operations. War changes fast enough.
I see. I think when you straightforwardly refuted
and later when you similarly disagreed with the ASI analogy (learning from pre-critical AI), I took that to mean that this ‘one-shotness’ concept was meant as more than simply ‘there is a critical try (meaning you can’t go back, and/because failure is approximately fatal)‘, but also to include ‘and you can not practically learn from relevantly-similar experience beforehand’. (On this definition, the asteroid case is ‘less one-shot’ than you’re classing ASI as because you can do relevantly similar practice beforehand if you have time, including perhaps on the very same asteroid, though with increasing, eventually critical stakes.)
But now I perceive that you mean ‘one-shotness’ to be the simpler thing, that there’s a critical try. And the essay was just additionally countering the putatively palliative ‘it will be OK though because we can learn beforehand’.
Ah, no, I now remember I was with you on the definition (see my ‘un-unpluggability’ comment), but I was noting (as you do in the essay) that the curse of distribution shift is an important adjunct, because the existence of a critical try is not in itself fatal (it might be easy, or you might have made it easy by practice). The asteroid case looks ‘easier’ to me, in that way, unless artificially constrained to be especially unexpected and rapid. cf Steve’s comment also discussing these related curses.