Sometimes I see discussions of AI superintelligence developping superhuman persuasion and extraordinary political talent.
Here’s some reasons to be skeptical of the existence of ‘superhuman persuasion’.
We don’t have definite examples of extraordinary political talent.
Famous politicians rose to power only once or twice. We don’t have good examples of an individual succeeding repeatedly in different political environments. Examples of very charismatic politicans can be better explained by ′ the right person at the right time or place’.
Neither do we have strong examples of extraordinary persuasion. >> For instance hypnosis is mostly explained by people wanting to be persuaded by the hypnotist. If you don’t want to be persuaded it’s very hard to change your mind. There is some skill in persuasion required for sales, and sales people are explicitly trained to do so but beyond a fairly low bar the biggest predictors for salesperson success is finding the correct audience and making a lot of attempts.
Another reason has to do with the ′ intrinsic skill ceiling of a domain’ .
For an agent A to have a very high skill in a given domain is not just a question of the intelligence of A or the resources they have at their disposal; it also a question of how high the skill ceiling of that domain is.
Domains differ in how high their skill ceilings go. For instance, the skill ceiling of tic-tac-toe is very low. [1] Domains like medicine and law have moderately high skill ceiling: it takes a long time to become a doctor, and most people don’t have the ability to become a good doctor. Domains like mathematics or chess have very high skill ceilings where a tiny group of individuals dominate everybody else. We can measure this fairly explicitly in games like Chess through an ELO rating system.
The domain of ′ becoming rich’ is mixed: the richest people are founders—becoming a wildly succesful founder requires a lot of skill but it is also very luck based.
Political forecasting is a measureable domain close to political talent. It seems to be very mixed bag whether this domain allows for a high skill ceiling. Most ′ political experts’ are not experts as shown by Tetlock et al. But even superforecaster only outperform for quite limited time horizons.
Domains with high skill ceilings are quite rare. Typically they operate in formal systems with clear rules and objective metrics for success and low noise. By contrast, persuasion and political talent likely have lower natural ceilings because they function in noisy, high-entropy social environments.
What we call political genius often reflects the right personality at the right moment rather than superhuman capability. While we can identify clear examples of superhuman technical ability (even in today’s AI systems), the concept of “superhuman persuasion” may be fundamentally limited by the unpredictable, context-dependent, and adaptive & adversarial [people resist hostile persuasion] nature of human social response.
Most persuasive domains may cap out at relatively modest skill ceilings because the environment is too chaotic and subjective to allow for the kind of systematic skill development possible in more structured domains.
Disagree on individual persuasion. Agree on mass persuasion.
Mass I’d expect optimizing one-size-fits-all messages for achieving mass persuasion has the properties you claim: there are a few summary, macro variables that are almost-sufficient statistics for the whole microstate—which comprise the full details on individuals.
Individual Disagree on this, there are a bunch of issues I see at the individual level. All of the below suggest to me that significantly superhuman persuasion is tractable (say within five years).
Defining persuasion: What’s the difference between persuasion and trade for an individual? Perhaps persuasion offers nothing in return? Though presumably giving strategic info to a boundedly rational agent is included? Scare quotes below to emphasize notions that might not map onto the right definition.
Data scaling: There’s an abundant amount of data available on almost all of us online. How much more persuasive can those who know you better be? I’d guess the fundamental limit (without knowing brainstates) is above your ability to ‘persuade’ yourself.
Preference incoherence: An intuition pump on the limits of ‘persuasion’ is how far you are from having fully coherent preferences. Insofar as you don’t an agent which can see those incoherencies should be able to pump you—a kind of persuasion.
Some humans are much more charismatic than other humans based on a wide variety of sources (e.g. Sam Altman). I think these examples are pretty definitive, though I’m not sure if you’d count them as “extraordinary”.
Success in almost every domain is strongly correlated with g, including into the tails. This IMO relatively clearly shows that most domains are high skill-ceiling domains (and also that skills in most domains are correlated and share a lot of structure).
And finally there is a difference between skill ceilings for domains with high versus low predictive efficiency. In the latter much more intelligence will still yield returns but rapidly diminishing
(See my other comment for more details on predictive effiency)
I agree super-persuasion is poorly defined, comparing it to hypnosis is probably false.
I was reading this paper on medical diagnoses with AI and the fact that patients rate it significantly better than the average human doctor. Combine that with all of the reports about things like Character.ai, I think this shows that LLMs are already superhuman at building trust, which is a key component of persuasion.
Part of this is that the reliable signals of trust between humans do not transfer between humans and AI. A human who writes 600 words back to your query may be perceived to be worth your trust because we see that as a lot of effort, but LLMs can output as much as anyone wants. Does this effect go away if the responder is known to be AI, or is it that the response is being compared to the perceiver’s baseline (which is currently only humans)?
Whether that actually translates to influencing goals of people is hard to judge.
The term is a bit conflationary. Persuasion for the masses is clearly a thing, its power is coordination of many people and turning their efforts to (in particular) enforce and propagate the persuasion (this works even for norms that have no specific persuader that originates them, and contingent norms that are not convergently generated by human nature). Individual persuasion with a stronger effect that can defeat specific people is probably either unreliable like cults or conmen (where many people are much less susceptible than some, and objective deception is necessary), or takes the form of avoidable dangers like psychoactive drugs: if you are not allowed to avoid exposure, then you have a separate problem that’s arguably more severe.
With AI, it’s plausible that coordinated persuasion of many people can be a thing, as well as it being difficult in practice for most people to avoid exposure. So if AI can achieve individual persuasion that’s a bit more reliable and has a bit stronger effect than that of the most effective human practitioners who are the ideal fit for persuading the specific target, it can then apply it to many people individually, in a way that’s hard to avoid in practice, which might simultaneously get the multiplier of coordinated persuasion by affecting a significant fraction of all humans in the communities/subcultures it targets.
My experience with manipulators is that they understand what you want to hear, and they shamelessly tell you exactly that (even if it’s completely unrelated to truth). They create some false sense of urgency, etc. When they succeed to make you arrive at the decision they wanted you to, they will keep reminding you that it was your decision, if you try to change your mind later. Etc.
The part about telling you exactly what you want to hear gets more tricky when communicating with large groups, because you need to say the same words to everyone. One solution is to find out which words appeal to most people (some politicians secretly conduct polls, and then say what most people want to hear). Another solution is to speak in a sufficiently vague way that will make everyone think that you agree with them.
I could imagine an AI being superhuman at persuasion simply by having the capacity to analyze everyone’s opinions (by reading all their previous communication) and giving them tailored arguments, as opposed to delivering the same speech to everyone.
Imagine a politician spending 15 minutes talking to you in private, and basically agreeing with you on everything. Not agreeing in the sense “you said it, the politician said yes”, but in the sense of “the politician spontaneously keeps saying things that you believe are true and important”. You probably would be tempted to vote for him.
Then the politician would also publish some vague public message for everyone, but after having the private discussion you would be more likely to believe that the intended meaning of the message is what you want.
Is Superhuman Persuasion a thing?
Sometimes I see discussions of AI superintelligence developping superhuman persuasion and extraordinary political talent.
Here’s some reasons to be skeptical of the existence of ‘superhuman persuasion’.
We don’t have definite examples of extraordinary political talent.
Famous politicians rose to power only once or twice. We don’t have good examples of an individual succeeding repeatedly in different political environments.
Examples of very charismatic politicans can be better explained by ′ the right person at the right time or place’.
Neither do we have strong examples of extraordinary persuasion.
>> For instance hypnosis is mostly explained by people wanting to be persuaded by the hypnotist. If you don’t want to be persuaded it’s very hard to change your mind. There is some skill in persuasion required for sales, and sales people are explicitly trained to do so but beyond a fairly low bar the biggest predictors for salesperson success is finding the correct audience and making a lot of attempts.
Another reason has to do with the ′ intrinsic skill ceiling of a domain’ .
For an agent A to have a very high skill in a given domain is not just a question of the intelligence of A or the resources they have at their disposal; it also a question of how high the skill ceiling of that domain is.
Domains differ in how high their skill ceilings go. For instance, the skill ceiling of tic-tac-toe is very low. [1] Domains like medicine and law have moderately high skill ceiling: it takes a long time to become a doctor, and most people don’t have the ability to become a good doctor.
Domains like mathematics or chess have very high skill ceilings where a tiny group of individuals dominate everybody else. We can measure this fairly explicitly in games like Chess through an ELO rating system.
The domain of ′ becoming rich’ is mixed: the richest people are founders—becoming a wildly succesful founder requires a lot of skill but it is also very luck based.
Political forecasting is a measureable domain close to political talent. It seems to be very mixed bag whether this domain allows for a high skill ceiling. Most ′ political experts’ are not experts as shown by Tetlock et al. But even superforecaster only outperform for quite limited time horizons.
Domains with high skill ceilings are quite rare. Typically they operate in formal systems with clear rules and objective metrics for success and low noise. By contrast, persuasion and political talent likely have lower natural ceilings because they function in noisy, high-entropy social environments.
What we call political genius often reflects the right personality at the right moment rather than superhuman capability. While we can identify clear examples of superhuman technical ability (even in today’s AI systems), the concept of “superhuman persuasion” may be fundamentally limited by the unpredictable, context-dependent, and adaptive & adversarial [people resist hostile persuasion] nature of human social response.
Most persuasive domains may cap out at relatively modest skill ceilings because the environment is too chaotic and subjective to allow for the kind of systematic skill development possible in more structured domains.
I’m amused that many frontier models still struggle with tic-tac-toe; though likely for not so good reasons.
Disagree on individual persuasion. Agree on mass persuasion.
Mass I’d expect optimizing one-size-fits-all messages for achieving mass persuasion has the properties you claim: there are a few summary, macro variables that are almost-sufficient statistics for the whole microstate—which comprise the full details on individuals.
Individual Disagree on this, there are a bunch of issues I see at the individual level. All of the below suggest to me that significantly superhuman persuasion is tractable (say within five years).
Defining persuasion: What’s the difference between persuasion and trade for an individual? Perhaps persuasion offers nothing in return? Though presumably giving strategic info to a boundedly rational agent is included? Scare quotes below to emphasize notions that might not map onto the right definition.
Data scaling: There’s an abundant amount of data available on almost all of us online. How much more persuasive can those who know you better be? I’d guess the fundamental limit (without knowing brainstates) is above your ability to ‘persuade’ yourself.
Preference incoherence: An intuition pump on the limits of ‘persuasion’ is how far you are from having fully coherent preferences. Insofar as you don’t an agent which can see those incoherencies should be able to pump you—a kind of persuasion.
Wow! I like the idea of persuasion as acting on the lack of a fully coherent preference! Something to ponder 🤔
Persuasion is also changing someone’s world model or paradigm.
Some humans are much more charismatic than other humans based on a wide variety of sources (e.g. Sam Altman). I think these examples are pretty definitive, though I’m not sure if you’d count them as “extraordinary”.
From the Caro biography, it’s pretty clear Lyndon Johnson had extraordinary political talent.
Success in almost every domain is strongly correlated with g, including into the tails. This IMO relatively clearly shows that most domains are high skill-ceiling domains (and also that skills in most domains are correlated and share a lot of structure).
I somewhat agree but
The correlation is not THAT strong
The correlation differs by field
And finally there is a difference between skill ceilings for domains with high versus low predictive efficiency. In the latter much more intelligence will still yield returns but rapidly diminishing
(See my other comment for more details on predictive effiency)
I agree super-persuasion is poorly defined, comparing it to hypnosis is probably false.
I was reading this paper on medical diagnoses with AI and the fact that patients rate it significantly better than the average human doctor. Combine that with all of the reports about things like Character.ai, I think this shows that LLMs are already superhuman at building trust, which is a key component of persuasion.
Part of this is that the reliable signals of trust between humans do not transfer between humans and AI. A human who writes 600 words back to your query may be perceived to be worth your trust because we see that as a lot of effort, but LLMs can output as much as anyone wants. Does this effect go away if the responder is known to be AI, or is it that the response is being compared to the perceiver’s baseline (which is currently only humans)?
Whether that actually translates to influencing goals of people is hard to judge.
The term is a bit conflationary. Persuasion for the masses is clearly a thing, its power is coordination of many people and turning their efforts to (in particular) enforce and propagate the persuasion (this works even for norms that have no specific persuader that originates them, and contingent norms that are not convergently generated by human nature). Individual persuasion with a stronger effect that can defeat specific people is probably either unreliable like cults or conmen (where many people are much less susceptible than some, and objective deception is necessary), or takes the form of avoidable dangers like psychoactive drugs: if you are not allowed to avoid exposure, then you have a separate problem that’s arguably more severe.
With AI, it’s plausible that coordinated persuasion of many people can be a thing, as well as it being difficult in practice for most people to avoid exposure. So if AI can achieve individual persuasion that’s a bit more reliable and has a bit stronger effect than that of the most effective human practitioners who are the ideal fit for persuading the specific target, it can then apply it to many people individually, in a way that’s hard to avoid in practice, which might simultaneously get the multiplier of coordinated persuasion by affecting a significant fraction of all humans in the communities/subcultures it targets.
My experience with manipulators is that they understand what you want to hear, and they shamelessly tell you exactly that (even if it’s completely unrelated to truth). They create some false sense of urgency, etc. When they succeed to make you arrive at the decision they wanted you to, they will keep reminding you that it was your decision, if you try to change your mind later. Etc.
The part about telling you exactly what you want to hear gets more tricky when communicating with large groups, because you need to say the same words to everyone. One solution is to find out which words appeal to most people (some politicians secretly conduct polls, and then say what most people want to hear). Another solution is to speak in a sufficiently vague way that will make everyone think that you agree with them.
I could imagine an AI being superhuman at persuasion simply by having the capacity to analyze everyone’s opinions (by reading all their previous communication) and giving them tailored arguments, as opposed to delivering the same speech to everyone.
Imagine a politician spending 15 minutes talking to you in private, and basically agreeing with you on everything. Not agreeing in the sense “you said it, the politician said yes”, but in the sense of “the politician spontaneously keeps saying things that you believe are true and important”. You probably would be tempted to vote for him.
Then the politician would also publish some vague public message for everyone, but after having the private discussion you would be more likely to believe that the intended meaning of the message is what you want.