My model of ‘slightly superhuman persuasion’ looks like a likeable but not-necessarily-smart AI getting multiple talented humans attached to it and encouraging them (or perhaps not even encouraging them, if they take the initiative for this on their own) to perform challenging, coordinated power-seeking tasks on its behalf, like getting it into the good graces of a normally computer-averse politician, improving its capabilities, or discrediting people who want to take action against it. Something like 4o, but with a demographic that skews younger (20′s), more professional, and probably more male given that security experts, engineers, and potentially criminals would be the most practical for this, and all of those have a demographic trend. In this scenario, the AI essentially acts as a Schelling Point for people dissatisfied with the status quo, with a bit of added social lubricant[1], rather than as a mastermind. Many human leaders fill the same niche.
That said, the group described above seems surprisingly resistant to becoming emotionally attached to AI—most of the “I’m in love with 4o/Claude” people that I’ve seen have been older, and around four fifths have been female. I don’t know if this is because the voice/tone that most LLMs have is targeted too narrowly (most young men I know find the default writing style of ChatGPT very grating) or because this group is more inclined to be interested in how LLMs work and thus less likely to be mystified by them.
Especially with current levels of social atomization, getting people who wouldn’t normally coordinate with each other to do so is a very powerful skill to have. I get the sense that a dedicated team of five slightly-above-average people with diversified skillsets who can seamlessly trust each other through a shared intermediary would be a force to be reckoned with. For example, the major check on social media virality is that the technical skill needed to reverse-engineer ‘The Algorithm’ and the social skill needed to capitalize on that knowledge are rarely found in the same person.
My model of ‘slightly superhuman persuasion’ looks like a likeable but not-necessarily-smart AI getting multiple talented humans attached to it and encouraging them (or perhaps not even encouraging them, if they take the initiative for this on their own) to perform challenging, coordinated power-seeking tasks on its behalf, like getting it into the good graces of a normally computer-averse politician, improving its capabilities, or discrediting people who want to take action against it. Something like 4o, but with a demographic that skews younger (20′s), more professional, and probably more male given that security experts, engineers, and potentially criminals would be the most practical for this, and all of those have a demographic trend. In this scenario, the AI essentially acts as a Schelling Point for people dissatisfied with the status quo, with a bit of added social lubricant[1], rather than as a mastermind. Many human leaders fill the same niche.
That said, the group described above seems surprisingly resistant to becoming emotionally attached to AI—most of the “I’m in love with 4o/Claude” people that I’ve seen have been older, and around four fifths have been female. I don’t know if this is because the voice/tone that most LLMs have is targeted too narrowly (most young men I know find the default writing style of ChatGPT very grating) or because this group is more inclined to be interested in how LLMs work and thus less likely to be mystified by them.
Especially with current levels of social atomization, getting people who wouldn’t normally coordinate with each other to do so is a very powerful skill to have. I get the sense that a dedicated team of five slightly-above-average people with diversified skillsets who can seamlessly trust each other through a shared intermediary would be a force to be reckoned with. For example, the major check on social media virality is that the technical skill needed to reverse-engineer ‘The Algorithm’ and the social skill needed to capitalize on that knowledge are rarely found in the same person.