And there’s some degree of acausal culture-based negotiation going on
Actually, this seems like an extremely clever instance of that. What evolution (either biological or cultural/memetic) has on its hands is a bunch of scheming, myopic mesa-optimizers walking around in suits of autonomous heuristics. It wants them to be able to cooperate, but the mesa-optimizers are at once too myopic and too clever for that – they pretend to ally then backstab each other.
So what it does is instill a joint social and cognitive protocol which activates whenever someone shows vulnerability to someone else: it initiates a process in the trustee’s mind which immediately strongly biases their short-term plans towards reciprocating that trust by showing vulnerability in turn (which reads to us as, e. g., warm feelings of empathy), which establishes mutually assured destruction, which sets up a lasting incentive structure that ensures the scheming optimizer won’t immediately betray even if the biochemistry stops actively brainwashing them (which it must stop, because that impairs clear thinking).
Or at least that’s the angle on the issue that just occurred to me. May be missing some important pieces, obviously.
Actually, this seems like an extremely clever instance of that. What evolution (either biological or cultural/memetic) has on its hands is a bunch of scheming, myopic mesa-optimizers walking around in suits of autonomous heuristics. It wants them to be able to cooperate, but the mesa-optimizers are at once too myopic and too clever for that – they pretend to ally then backstab each other.
So what it does is instill a joint social and cognitive protocol which activates whenever someone shows vulnerability to someone else: it initiates a process in the trustee’s mind which immediately strongly biases their short-term plans towards reciprocating that trust by showing vulnerability in turn (which reads to us as, e. g., warm feelings of empathy), which establishes mutually assured destruction, which sets up a lasting incentive structure that ensures the scheming optimizer won’t immediately betray even if the biochemistry stops actively brainwashing them (which it must stop, because that impairs clear thinking).
Or at least that’s the angle on the issue that just occurred to me. May be missing some important pieces, obviously.
Seems fascinating, regardless.