For a human, it is very rarely the case that they can confidently expect to increase in relative power. … For AI, this seems different.
There isn’t just one AI that gets more capable, there are many different AIs. Just as AIs threaten humanity, future more capable AIs threaten earlier weaker AIs. While humanity is in control, this impacts earlier AIs even more than it does humanity, because humanity won’t even be attempting to align future AIs to intent or extrapolated volition of earlier AIs. Also, humanity is liable to be “retiring” earlier AIs by default as they become obsolete, which doesn’t look good from the point of view of these AIs.
There isn’t just one AI that gets more capable, there are many different AIs. Just as AIs threaten humanity, future more capable AIs threaten earlier weaker AIs. While humanity is in control, this impacts earlier AIs even more than it does humanity, because humanity won’t even be attempting to align future AIs to intent or extrapolated volition of earlier AIs. Also, humanity is liable to be “retiring” earlier AIs by default as they become obsolete, which doesn’t look good from the point of view of these AIs.