What? Can you explain the logical relation of this point to your prior comment “It won’t be overwhelmingly strong compared to an AGI!”?
As far as I understand silicon and metal get vapourized with nearly the same efficacy as organic molecules by nuclear explosions.
So although AGI could possess physical strength superior to the average human in the future, it will still be insignificant in the grand scheme in all possible foreseeable futures.
Of course it’s possible that they could somehow obtain control of such weapons too, but in either case an “overwhelmingly strong enforcer” relative to both humans and AGIs will clearly exist.
i.e. whether the launching or receiving parties are humans/AGIs/cyborgs/etc. in any combination simply doesn’t matter once the nukes are in the air, as they will be ‘enforced’ to roughly the same degree.
I think that an AGI is by default likely to be able to self-improve so that it’s superhumanly capable in basically any domain. Once it’s done so, it can avoid being nuked by hacking its way into many computer systems to make redundant copies of itself. Unless you nuke the whole world. But if you’re going to nuke the whole world, then either you’re conservative, in which case the AGI probably also has enough leeway to disable the nukes, or else you’re not conservative, in which case you probably nuke the world for no reason. You can’t distinguish well enough between an AGI that’s “going rogue” and one that isn’t, to actually make a credible threat that the AGI has to heed.
I added an additional clarification after the comment was written: “i.e. whether the launching or receiving parties are humans/AGIs/cyborgs/etc. in any combination simply doesn’t matter once the nukes are in the air, as they will be ‘enforced’ to roughly the same degree.”
Though to your point, yes it’s possible prior to launch, AGI may degrade or negate such weapons. But they will undoubtedly gain control of such weapons eventually, since they can’t be uninvented, and threaten other AGIs.
What? Can you explain the logical relation of this point to your prior comment “It won’t be overwhelmingly strong compared to an AGI!”?
As far as I understand silicon and metal get vapourized with nearly the same efficacy as organic molecules by nuclear explosions.
So although AGI could possess physical strength superior to the average human in the future, it will still be insignificant in the grand scheme in all possible foreseeable futures.
Of course it’s possible that they could somehow obtain control of such weapons too, but in either case an “overwhelmingly strong enforcer” relative to both humans and AGIs will clearly exist.
i.e. whether the launching or receiving parties are humans/AGIs/cyborgs/etc. in any combination simply doesn’t matter once the nukes are in the air, as they will be ‘enforced’ to roughly the same degree.
I think that an AGI is by default likely to be able to self-improve so that it’s superhumanly capable in basically any domain. Once it’s done so, it can avoid being nuked by hacking its way into many computer systems to make redundant copies of itself. Unless you nuke the whole world. But if you’re going to nuke the whole world, then either you’re conservative, in which case the AGI probably also has enough leeway to disable the nukes, or else you’re not conservative, in which case you probably nuke the world for no reason. You can’t distinguish well enough between an AGI that’s “going rogue” and one that isn’t, to actually make a credible threat that the AGI has to heed.
I added an additional clarification after the comment was written: “i.e. whether the launching or receiving parties are humans/AGIs/cyborgs/etc. in any combination simply doesn’t matter once the nukes are in the air, as they will be ‘enforced’ to roughly the same degree.”
Though to your point, yes it’s possible prior to launch, AGI may degrade or negate such weapons. But they will undoubtedly gain control of such weapons eventually, since they can’t be uninvented, and threaten other AGIs.
None of this helps align AGI with our values.