For a threat to be significant, it has to be believed. In the case of AGI, this probably means the AGI itself being unable to renege on the threat. If two such met, wouldn’t fighting be inevitable? If so, how do we know it wouldn’t be worthwhile for at least some AGIs to make such a threat, sometimes?
Then again, ‘Maintain control of my current level of resources’ could be a schelling point that prevents descent into conflict.
But it’s not obvious why an AGI would choose to draw their line in the sand their though, when ‘current resources plus epsilon% of the commons’ is available. The main use of schelling points in human games is to create a more plausible threat, whereas an AGI could just show its source code.
For a threat to be significant, it has to be believed. In the case of AGI, this probably means the AGI itself being unable to renege on the threat. If two such met, wouldn’t fighting be inevitable? If so, how do we know it wouldn’t be worthwhile for at least some AGIs to make such a threat, sometimes?
Then again, ‘Maintain control of my current level of resources’ could be a schelling point that prevents descent into conflict.
But it’s not obvious why an AGI would choose to draw their line in the sand their though, when ‘current resources plus epsilon% of the commons’ is available. The main use of schelling points in human games is to create a more plausible threat, whereas an AGI could just show its source code.
An AGI won’t turn itself into a defecting rock, when there is a possibility of pareto improvement over that.