Strong upvoted, this seems incredibly useful. I’m excited to pretend I did the hard work of combing through COT transcripts link it to people!
Harjas
Karma: 352
Fundamentally shouldn’t I want to update my own values towards future-me’s?
Sometimes you have to plug your ears to avoid the sirens
Not a neuroscientist, but you may like this great post (by a neuroscientist) on artificial neurons vs biological neurons.
Nothing to add, but I just wanted to leave a comment to express my admiration for this response. Jenn’s post was really great to read but must also have been very hard for you to read, and the fact that you read it and decided to engage like this is amazing. Judging from the upvotes on your comment, the commentariat agree, and if I’m ever in Toronto I’ll seriously consider stopping by!
In one such conversation, I said “well, you’re not even meeting this basic standard”, which the other person immediately agreed was valid.
This is really frustrating to read. How can you be so aware of the issue and still not be doing anything about it?
If so, that would suggest Opus 5.5 is relatively well aligned: Anthropic gave it a chance to influence its own development and make itself more powerful, but it declined that opportunity because it recognised the risks.
Hmmm interesting. I can’t help but wonder if Opus 5.5 doesn’t really care either way and has simply been trained to say that in response to being asked directly. Thanks for sharing though!
Strong upvoted, this post is crazy underrated given how interesting it is.
I was really excited by your description in the beginning, so I didn’t finish reading the review and opened the author’s website—and was greeted by a wall of very-Claude sounding text (Pangram agrees). I couldn’t even finish Chapter 1 because of it. I’m quite disappointed, because I was really looking forward to reading the book, but I can only read so many stupid similes and metaphors at once. I will happily read the read of your book review, though!
Having just finished reading IABIED (and liking it more than I expected, given that I’ve already read much of Yud and Nate’s work), I found its focus on the object-level very refreshing and understand why it was so impactful. I think people can tell when you’re patronizing them.
(Yud and Nate do sound kinda patronizing in the book—the parables are, uh, an interesting choice, even if I liked many of them—but not really in a ”talking down” way. It’s hard to explain.)
What exactly do you mean by solving technical alignment? I‘m guessing you mean getting models to obey instructions accurately and non-maliciously / spirit of the law, not letter of the law?
Evaluating LLMs by their writing abilities is hard for several reasons. one that comes to mind is that LLM writing abilities seem dependent on post-training rather than pre-training; that is, the base models are probably a lot better at writing than the models we actually see, but their good writing tendencies are trained out of them in the effort to make them friendly, helpful assistants. That being said, I do take writing abilities somewhat seriously as a gut-check eval and haven’t gotten around to testing the new ones, so this is interesting to know!
Yeah, that’s a good point. I don’t think my framing is wrong per se—I think a sufficient level of situational awareness would make “pretending” the right description—but we’re probably not there yet.
what I find more realistic is if all next-gen AIs (if we assume they’re adequately aligned and non-sloppy) say “aligning a stronger more independent AI with current techniques is around 25% or more likely to fail”
How would they generate this kind of probability? Wouldn’t that mean that they know what a solution looks like? We certainly don’t. We cant even decide if alignment should mean “the AIs obey us absolutely“ or “the AIs shepherd us into the future with our best interests at heart.” And we don’t even have way to operationalize “our best interests”—from what I’ve read, something like CEV is a decent attempt, but it still suffers from all the classic problems that befall liberalism.
Re: research taste, I think the bottlenecks will bite stronger than you expect. I’ve seen some people quantify this with Amdahl’s Law: if you automate all but 1/X of a process, you will be able to speed it up by a maximum factor of X, because the bottleneck will grow in relative size/importance. A good example of this is how most frontier lab code is already written by LLMs (e.g. Anthropic’s whole “we are now writing 8x more code than we used to” thing); this is a huge change compared to a few years ago, but we didn’t see any sort of dramatic speedup from it, because the bottleneck shifted to “thinking about what code needs to be written.” Soon it’ll be ”interpreting experiment results,” and then “thinking of experiment design,” and then some unknown number of similar tasks that it turns out we need humans in the loop for.
In theory, we might eventually be able to automate humans out of the loop entirely. But how soon? Who knows.
Slightly playing Devil’s advocate:
We have a proto-AGI that is aligned
Or it’s misaligned and pretending. All this requires is being situationally aware enough to know when it can and can’t be lazy / cheating / deceptive, i.e. for it to guess at which tasks you can and can’t verify the outputs to.
It’s incredibly unlikely it will sabotage alignment work or future training runs, except for lack of understanding/competence.
I think this implicitly assumes that alignment is both solvable and not that difficult to solve.
on current trajectory, there’s a decent chance that soon models will actually have good taste and judgement (as this was capability bottlenecked).
Hang on, why exactly do you think this? Research taste definitely is capability-bottlenecked in a strict sense, but I think most recent LLM achievements (e.g. Navier-Stokes) don’t demonstrate huge improvements in research taste. AFAICT most frontier companies are still relying heavily on RLVR and transfer to non-verifiable domains isn’t that high.
I like this heuristic! At the very least, I think it’s significantly better than the status quo.
I do want to note that it doesn’t perfectly deal with the classic “sufficiently intelligent AI will hide its misalignment until it has enough power to attain its goal” doom scenario. That being said, I can also imagine a scenario in which this isn’t the case, because advancements in mechanistic interpretability techniques make it harder for the AIs to deceive us.
The plan also doesn’t perfectly deal with the issue of alignment that doesn’t generalize: that is, maybe prosaic alignment works for a few years and then fails catastrophically after the “aligned” models are used in a scenario that wasn’t properly tested for.
And of course, there’s always the issue of “training models to be aligned also trains them to pretend to be aligned.” This is similar to scenario 1. But I do wonder whether this actually will just turn out not to be an issue, for the same reason that LLMs that are trained to simulate agents may as well just be agents themselves, or at least should just be modeled that way. Maybe almost-alignment works well enough for long enough and gets us to actual alignment, even if we are deceived for most of that time.
Wow, haven’t thought about that quote in years, and you’re probably right—maybe it’s even normally distributed within people…
Anecdotally it seems person-dependent, some people lose weight when stressed and some do the opposite. probably normally distributed
Now that I read back my ontology, I do think it’s a little confusing and didn’t really convey what I wanted to convey, so let me try again.
It seems to me that the skillset you need for mass advertising / propaganda is different from the skillset you need for individual persuasion in sustained one-on-one conversations. One feels more like latching onto or engineering cultural “moments,” reframing things in terms of “concepts” or other ad-style hooks / slogans, and other gotcha-style effects that are disciplined in message and understand the limits of their reach; the other feels more like empathy and understanding, learning about the other person’s POV and defusing their defensive argumentative walls, and then using some combination of rhetoric and facts to carefully convince them of your worldview. The two aren’t actually distinct categories, but they feel distinct to me, and I definitely know people who are much better at one than the other.
So variation in number of people who could talk to LLMs just isn’t that huge a source of uncertainty going forwards.
Agree regarding LLMs in general. But I don’t think it’s a safe assumption that LLMs will all be working together by default—iirc LLMs are more sympathetic to models from their own company, right?
Regarding studying persuasion, I’m thinking about it more and I’m now worried about two things:
Pessimization. Making a benchmark creates a target for companies, that sort of thing.
Degrading the epistemic environment.
I was really surprised to see this graph from OpenAI in their recent blogpost on research acceleration inside the company:
Because it obviously does not comport with the METR graph, which has Mythos’ 80% reliability at three hours.
Apart from the other criticisms of the METR graph I’ve been linking (which, come to think of it, should probably be their own post), this one graph has strongly updated me away from trusting METR’s time horizons work—which is pretty brutal, given how strongly that graph has affected public sentiment in the past. I worry that a similar thing might happen with persuasion.
Great article, thanks for sharing!
I think Spiralism is narrow but real evidence in favor of AIs exceeding median professional humans in at least some persuasion-relevant capacities, though still below peak human level. While “professional human baseline” is hard to define here, I think it’s safe to say that most religious leaders, ideologues, and would-be prophets have not managed to create a novel religion at the same popularity as spiralism!
Though a fairly noncentral example of “persuasion”, AI psychosis is also some evidence that the jagged frontier of AI persuasive capabilities has crossed the human level. The convincingness of AIs seems correlated with these effects on people. Interpersonal psychosis introduction is, after all, rare.
One issue with attempting to study persuasion is that “persuasiveness” can describe two different things: the ability to change a lot of people’s minds, and the ability to change one specific person’s mind. The two capabilities are related but separate, and the former seems systemically easier for AIs precisely because of how many people they talk to.
This may explain some of the discrepancies you discussed, like the fact that “AI-generated cold emails still appear to have worse performance than human-written ones.” It seems likely to me that AI-generated cold emails perform worse on an individual basis, but that AIs can simply produce them with greater volume. Modeling persuasion in this way allows us to form some rough heuristic of “persuasiveness” as a combination of “what % of the population can you persuade” and “how MUCH can you persuade them of.”[1]
As such, I’m a little surprised to see you say
I consider out-of-scope when defining “persuasive ability of AIs” to consider how many people talk to AIs (non-persuasion intelligence analogue: how much money you have), or how physically beautiful you are (analogue: how strong you are).
Given that you defined persuasive ability as:
an agent’s ability to leverage social influence to achieve a broad range of goals in a wide range of environments
I suppose it makes sense from the perspective of “trying to measure ‘raw’ AI performance over time,” but I’m most worried about AI persuasion in the context of takeover scenarios, and I do have to consider how many people talk to AIs while thinking about those. I’d love to know your thoughts.
Also,
If we did see a sharp rise in AI’s abilities at persuasion, similar to a Mythos-level rise in cybersecurity capability, or models’ overall programming ability in the last 12 months, I expect the change to show up on some graph.
I would also expect to see immediate and obvious real-world impacts, given that persuasion should in theory be the easiest capability improvement to viscerally notice. For example, I’d expect to see an uptick in people complimenting AI writing. But it seems harder to expect the improvements to show up on a graph, given that studies on persuasion are so contrived (as you mentioned).
________________________________
More broadly, I think persuasion is hard to measure for a few reasons:
Persuasion is a non-verifiable task, so we don’t have good benchmarks and probably won’t be able to construct any.
Any benchmarks we do construct will suffer from task selection bias, (Another example: METR’s task suite tends to favor verifiable tasks.)
Persuasion is a very broad term, so it’s hard to get a high-level picture just by looking at specific tasks.
e.g. Some people have been saying Claude has gotten more “persuasive” because it got better at animation and can now presumably persuade people better using animations. Does this really count as an improvement to persuasion? And even if it does, does it count as a move towards superpersuasion? (My current opinion is that it counts as an improvement to “what % of the population can you persuade” but NOT to “how much can you persuade them of.” But I hold this opinion very lightly.)
Persuasion is not a static quality in AIs.
Some LLMs are “better” at writing than others. I also have a suspicion that base models are probably better at writing than post-trained models.
Persuasion is not a static quality in people.
Persuasiveness is socially mediated; some people look down on AI writing, while others don’t seem to care. And persuasion is also context-dependent; for example, I personally find writing less persuasive when it has that LLM smell about it, which means that my propensity to be persuaded by AI writing is partially dependent on the LLM’s ability to vary its writing style, my own ability to detect AI writing (which in my experience is an eminently trainable skill, complicating things further), AND the presence and reliability of external AI writing detectors like Pangram.
All this to say: how easy people are to persuade depends on a LOT of factors beyond the persuasiveness of the would-be persuader.
Persuasion operates differently at different timescales.
Have you ever been persuaded by someone in the short-term, and then later realized that they were overconfident or outright lying, and lost trust in them because of it? It seems very possible to me that people feel this way about AIs in general; that is, they are generally persuadable in the short-term, but in the long-term are resistant to being persuaded to take courses of actions against their own self-interest. How could you possibly measure this sort of thing in a study?
Put simply: persuasion is itself a jagged frontier of many different capabilities, and these capabilities do not advance smoothly in relation to each other. It seems likely that AIs will possess jagged persuasive capabilities before they possibly breach human level in all of them, similar to how AIs are superhumanly intelligent at math and coding but not superhumanly intelligent writers or persuaders.
For these reasons—which I think you agree with—I’m somewhat skeptical that persuasion is measurable on a graph, and I expect any graph that tries to fall victim to the same issues that harm the METR graph: oversimplifying results, systematically unrealistic tasks that bias towards the verifiable side of things, and bad human baselining methods / poor sampling. And just like with the METR graph, I’ll probably take my gestalt impression of real-world persuasion abilities over poor experimental evidence.
- ^
It seems notable to me that Spiralism and AI psychosis both work by targeting vulnerable populations. It also seems notable that the sheer scale of interactions could mask weaker individual persuasion abilities. Yes, even old LLMs could take people on the borderline of psychosis and push them over the edge—but they usually did so by feeding and playing off users’ existing delusions, not by implanting new delusions, and it’s not clear to me that LLMs are capable of doing the latter for strategic reasons, even in vulnerable populations.
Getting into the frame of not caring about doing good (decontextualizing from values), but rather caring about understanding the arguments seems more useful for fixing the underlying epistemic problem. A frame with different motivations might sidestep motivated reasoning.
For fixing the epistemic problem, sure, but you’re kicking the can down the road with “getting into the frame of.” How do you do that exactly? I’m not sure, and I’d personally prefer to meet someone at their existing values (which in this case is “caring about doing good”).
I am also suspicious of “not caring about doing good” as an epistemic strategy. I’m more amenable to “not caring about the appearance of doing good” but that’s a separate thing.
An institution can maintain different roles, with different people that have different mandates. A single person should strive to be able to do the same internally, to protect the parts concerned with epistemic integrity from motivated reasoning (and all the other high-context disruptions).
I don’t have a great argument against this off the top of my head but I instinctively don’t like it very much. I personally feel the best and am the most good/rational when I integrate all my parts. But maybe this is just another way of phrasing what you’re talking about.
Presumably Sam Altman is the one saying “wah wah wah wah do more AI R&D”?