How plausible is it that there is an offense-defence balance? Imagine that there is a metric of the ability to overwrite someone else’s goals, to protect one’s goals from being overwritten and that the latter metric increases with intelligence faster than the former. Then a collective of agents with a high intelligence would be immune to such attacks, while a collective of agents with low intelligence would be vulnerable to epidemics where a subgoal spreads to the whole collective. How similar is the latter to failure modes present in human collectives?
How plausible is it that there is an offense-defence balance? Imagine that there is a metric of the ability to overwrite someone else’s goals, to protect one’s goals from being overwritten and that the latter metric increases with intelligence faster than the former. Then a collective of agents with a high intelligence would be immune to such attacks, while a collective of agents with low intelligence would be vulnerable to epidemics where a subgoal spreads to the whole collective. How similar is the latter to failure modes present in human collectives?