Hmm, when I imagine “Scheming AI that is not easy to shut down with concerted effort, are trying to sabotage operations, but are weak enough such that you can bargain/negotiate with them” I can imagine this outcome inspiring a lot more caution relative to many other worlds where control techniques work well but we can’t get any convincing demos/evidence to inspire caution (especially if control techniques inspire overconfidence).
But the ‘is currently working on becoming more powerful’ part of your statement does carry a lot of weight.
Hmm, when I imagine “Scheming AI that is not easy to shut down with concerted effort, are trying to sabotage operations, but are weak enough such that you can bargain/negotiate with them” I can imagine this outcome inspiring a lot more caution relative to many other worlds where control techniques work well but we can’t get any convincing demos/evidence to inspire caution (especially if control techniques inspire overconfidence).
But the ‘is currently working on becoming more powerful’ part of your statement does carry a lot of weight.
People will sure be scared of AI, but the arms race pressure will be very strong, and I think that is a bigger consideration