Some vague idea: Alignment can be fragile. Can capabilities be made fragile too?
I think fragile capabilities can be potentially useful in situations that needs to prevent tampering the model, eg finetuning a model to jailbreak / learn dangerous bioweapon capabilities.
For the banning of these weapons, how much does effectiveness weigh against moral concerns? If usefulness weighs a lot, then these examples won’t generalize to TAI.
Unless there are very clear, convincing evidence that TAI isn’t controllable with current paradigm, then it will still be perceived as a highly useful tech. (Even if such evidence exists, IMO there’s high possibility that they’ll just cope harder.)
Biochemical weapons: These are only useful against civilians and pre-modern armies. Modern armies can easily afford equipments to protect against these.
https://acoup.blog/2020/03/20/collections-why-dont-we-use-chemical-weapons-anymore/
(I saw this article mentioned somewhere in another LW post. When I see TsviBT’s shortform I immediately recalled this article, so I wrote this post.)
Space nukes and LEO missiles: In space there’s no cover, they’re easily detectible. Without air, dodging maneuver cost significant dv. This means overall less survivability than ground / sea based nukes.
Deploying missiles in LEO also requires a more complicated trajectory than traditional ground / sea based missiles, which cost more dv. If they need to stay in space for a long time, then reliability and maintainence also becomes a serious problem.
Obedience in this post looks like corrigibility which has been discussed a lot.
Zachtronics games does part of that. In those games the player doesn’t do the tasks directly, instead they need to program a bot (or other systems) to do the task. While mirroring the player is impossible, it should be possible to mirror the bots programmed by the player.
While price gouging can quickly mobilize forces to satisfy the emergency demand, they can also have problematic second-order effects. If a price gouge is too high, then this allows certain agents to benefit from the disaster. This creates a perverse incentive that disincentivizes disaster prevention, and potentially even incentivizing artifically intensifying / creating disasters.