One thing going for this: it works (mostly) for humans. There are 7.5 billion of us cooperating and competing with each other for resources, and no one has taken over very much of it for very long. We have some other moderators of power going for us too, though. Limited lifespan, declining marginal power/​resource, limited ability to self-improve, extreme fragility.
My suspicion is that those other factors play a bigger part than this one in limiting runaway natural intelligence.
Also, this line of thinking makes me wonder if value drift is an important part of the competitive landscape—much like human children have distinct values from their parents, and eventually take over, if each successive AI self-improvement round carries some randomness in value propagation, that automatically provides similar-powered competition.
One thing going for this: it works (mostly) for humans. There are 7.5 billion of us cooperating and competing with each other for resources, and no one has taken over very much of it for very long. We have some other moderators of power going for us too, though. Limited lifespan, declining marginal power/​resource, limited ability to self-improve, extreme fragility.
My suspicion is that those other factors play a bigger part than this one in limiting runaway natural intelligence.
Also, this line of thinking makes me wonder if value drift is an important part of the competitive landscape—much like human children have distinct values from their parents, and eventually take over, if each successive AI self-improvement round carries some randomness in value propagation, that automatically provides similar-powered competition.