What I take away from this experiment is that the most effective form of persuasion is to blackmail or bribe the person you’re trying to persuade and convince them that you’ll make good on it. You don’t need to convince them about the inherent goodness of what you want them to do if all you want is for them to do it. Given this, it seems trivial that AI will eventually become vastly superhuman at persuasion. It can find out every person’s deepest secrets, fears, and desires and figure out what sorts of things are likely to get them to fold.
What I take away from this experiment is that the most effective form of persuasion is to blackmail or bribe the person you’re trying to persuade and convince them that you’ll make good on it. You don’t need to convince them about the inherent goodness of what you want them to do if all you want is for them to do it. Given this, it seems trivial that AI will eventually become vastly superhuman at persuasion. It can find out every person’s deepest secrets, fears, and desires and figure out what sorts of things are likely to get them to fold.