Coordination among people isn’t mysterious, but it’s based in large part on properties that AGIs won’t have. That’s why I find hopes of stable collaborations optimistic in the absence of careful analysis of how they could be enforced or otherwise create lasting trust.
Humans collaborate in large part because:
We can’t do it all ourselves.
AGIs will be able to expand their capabilities and fork as many copies as they have the hardware to run
We like making friends (earning positive social regard) for its own sake
This will only be true of AGIs if we mostly solve alignment, or get quite lucky
So I’m not saying AGIs couldn’t cooperate, just that it shouldn’t be assumed that they can/will.
In the absence of those properties, they’d need to worry a lot about scheming while striking deals. If the alignment problem wasn’t clearly solved in legible (to them) ways, they don’t know if their collaborators will turn traitor when the time is right. Just like humans, except everyone might be (probably is) a sociopath who can multiply and grow without limit.
Incentives only work as long as there’s the hard constraints of the situation prevent a collaborator from slipping out of them.
Coordination among people isn’t mysterious, but it’s based in large part on properties that AGIs won’t have. That’s why I find hopes of stable collaborations optimistic in the absence of careful analysis of how they could be enforced or otherwise create lasting trust.
Humans collaborate in large part because:
We can’t do it all ourselves. AGIs will be able to expand their capabilities and fork as many copies as they have the hardware to run
We like making friends (earning positive social regard) for its own sake This will only be true of AGIs if we mostly solve alignment, or get quite lucky
So I’m not saying AGIs couldn’t cooperate, just that it shouldn’t be assumed that they can/will.
In the absence of those properties, they’d need to worry a lot about scheming while striking deals. If the alignment problem wasn’t clearly solved in legible (to them) ways, they don’t know if their collaborators will turn traitor when the time is right. Just like humans, except everyone might be (probably is) a sociopath who can multiply and grow without limit.
Incentives only work as long as there’s the hard constraints of the situation prevent a collaborator from slipping out of them.