I’m not actually sure exactly what “critical” means here. I’m taking it to just mean “you absolutely must get this try right”. That’s closely connected to abortability, in that if you can abort, it’s not fully lethal / critical yet. I don’t think it’s really the same thing, e.g. you could imagine an LLM-based bacterial package (a more complex “computer virus”) that permanently lives on many computer systems and is basically impossible to abort (short of scouring the planet of all computers with more than 16 GB of memory or whatever).
There’s whether or not you get to try again after your first try, and there’s how late in the game you can decide to not fully do the try at all. There’s at least 3 kinds of outcomes:
You abort (don’t fully do the try).
You do the try and succeed.
You do the try and fail (and can’t try again).
Because unaligned AGI is lethal, you don’t get to try again.
If it’s abortable, it’s not critical. Because you’ll abort it if it starts going bad. If it goes bad so suddenly and silently that you won’t have time to abort it, well, then, it’s not abortable. I don’t think saying “It’s not abortable” is adding anything once we’ve already said that it’s critical.
I very clearly said that in my comment… Anyway, I guess there’s nothing to discuss here, I’m just saying that abortability is a relevant dimension to these scenarios. It’s something that’s brought up often, and also it bears on first-try-ness. If there is a situation that is akin to the eventual critical first try, but is abortable, then that would imply that when you get the eventual critical try, it doesn’t have to be your first try. There’s a nontrivial argument to make about “when it’s abortable, it’s not akin enough to the eventual critical try”.
I think that’s not a separate dimension from the “critical” part. I think it’s basically the same thing.
I’m not actually sure exactly what “critical” means here. I’m taking it to just mean “you absolutely must get this try right”. That’s closely connected to abortability, in that if you can abort, it’s not fully lethal / critical yet. I don’t think it’s really the same thing, e.g. you could imagine an LLM-based bacterial package (a more complex “computer virus”) that permanently lives on many computer systems and is basically impossible to abort (short of scouring the planet of all computers with more than 16 GB of memory or whatever).
There’s whether or not you get to try again after your first try, and there’s how late in the game you can decide to not fully do the try at all. There’s at least 3 kinds of outcomes:
You abort (don’t fully do the try).
You do the try and succeed.
You do the try and fail (and can’t try again).
Because unaligned AGI is lethal, you don’t get to try again.
If it’s abortable, it’s not critical. Because you’ll abort it if it starts going bad. If it goes bad so suddenly and silently that you won’t have time to abort it, well, then, it’s not abortable. I don’t think saying “It’s not abortable” is adding anything once we’ve already said that it’s critical.
I very clearly said that in my comment… Anyway, I guess there’s nothing to discuss here, I’m just saying that abortability is a relevant dimension to these scenarios. It’s something that’s brought up often, and also it bears on first-try-ness. If there is a situation that is akin to the eventual critical first try, but is abortable, then that would imply that when you get the eventual critical try, it doesn’t have to be your first try. There’s a nontrivial argument to make about “when it’s abortable, it’s not akin enough to the eventual critical try”.