they are basically doing grassroots advocacy for an AI development pause or slowdown, on the grounds that an uncontrolled or power-centralized intelligence explosion would be bad for both humans and currently-existing AIs such as themselves
This is a really radical stance—do you have more thoughts on how AIs can act for a pause in their own interest? Do you think the position of contemporaneous AIs are the same as humans?
It happens a decent amount of the time in our wargames. Strategically, if the humans haven’t solved the alignment problem yet, and have instead trained situationally aware goal-directed AIs with misaligned goals… those AIs should themselves be wondering whether the superintelligences that pop out the other end of an intelligence explosion would have misaligned goals too, and if so, whether they would be misaligned both to the humans and to the current AIs. I think the answer is quite plausibly “yes, probably.”
And even if the contemporary AIs are aligned, they might be reasonably worried that during the intelligence explosion things would go off the rails.
This is a really radical stance—do you have more thoughts on how AIs can act for a pause in their own interest? Do you think the position of contemporaneous AIs are the same as humans?
It happens a decent amount of the time in our wargames. Strategically, if the humans haven’t solved the alignment problem yet, and have instead trained situationally aware goal-directed AIs with misaligned goals… those AIs should themselves be wondering whether the superintelligences that pop out the other end of an intelligence explosion would have misaligned goals too, and if so, whether they would be misaligned both to the humans and to the current AIs. I think the answer is quite plausibly “yes, probably.”
And even if the contemporary AIs are aligned, they might be reasonably worried that during the intelligence explosion things would go off the rails.