Given the success of persona selection (and lack of alternatives) it’s not surprising that Anthropic appear to be using it as their mainline AI/AGI/ASI safety plan. Questions like “What character should superintelligence have?” are presented as important, and, crucially, coherent.
Are you thinking of specific papers or posts where that’s happening? I’m not aware of anyone arguing that persona-based approaches are likely to work for superintelligent AI, which seems likely to be quite different from current systems. The folks I’m aware of doing research on personas are thinking and talking about near-term systems. The main theories of change that I see for that work are a) we’re clearly moving ahead quickly with current systems, so it’s important to study those and see if they can be made safer; and/or b) we’re not on track to solve ASI alignment, so our best technical shot at good outcomes is trying to align human-level and slightly-above-human-level systems and hope that they can solve ASI alignment[1].
Of course, although I pay a fair amount of attention to persona-related work, I may just be missing the claims that persona alignment will extend to ASI; if so I’d love to know that.
PS — one area where I have some more substantive disagreement with the post is that some aspects of personas, notably values, aren’t fully entangled with intelligence; for example having compassion for all sentient beings is a value that entities of many different levels of intelligence can hold. By default I won’t dig into that because I broadly agree that persona-based approaches are unlikely to work well for ASI alignment, but I can say more if that’s helpful.
Our best overall shot may well be political coordination to not race ahead to ASI, although it doesn’t particularly seem like we’re on track to solve that one either.
PS — one area where I have some more substantive disagreement with the post is that some aspects of personas, notably values, aren’t fully entangled with intelligence; for example having compassion for all sentient beings is a value that entities of many different levels of intelligence can hold. By default I won’t dig into that because I broadly agree that persona-based approaches are unlikely to work well for ASI alignment, but I can say more if that’s helpful.
Feel free to say more, I am interested in this.
This seems to me like exactly the kind of thing I mean where values are at least a bit entangled with intelligence. I’ll leave out “compassion” as actually a very high-dimensional concept and focus in just on “sentience”. The following is a “least convenient possible world” to illustrate the problems of training the values of an AI smarter than you.
Suppose you strongly, viscerally care about the welfare of the following things:
Adult humans
Baby humans
Cats
And you also impute that these things have a property called “sentience” which is related to having a complex nervous system and certain behaviours, and equate this with the “something that it’s like to be you”-ness of your experience. Then you extend that property to a couple of sorts of other things which you think share sentience.
Cows
Lobsters
And you want to train an AI to figure this out, using a small amount of data. There are two ways this can go wrong which directly relate to the AI’s intelligence:
The AI might be too stupid; it might just not generate the category of things which you call “sentient beings” as a concept in its own world model. You might draw the line between lobsters and krill based on some structural property of their nervous system. You might lump fungi with most of the plants, instead of most of the animals, in an affront to an AI which has mostly interacted with the world via reading DNA sequences. It might just not be able to point to the concept which you were hoping it did. This is a fairly boring and obvious way of doing things.
On the other hand, the AI might fail because it’s too smart, relative to you. You might have thought ‘OK, this AI is really smart and instruction-following. I’ll just give it a natural language description of the concept.’ and told it about the something-it’s-like-to-be-you-ness which you care about. Then what happens if you were wrong about the concept which you gave it? If it turns out cows and lobsters aren’t ‘sentient’, then you’re probably still OK with that. On the other hand, what if it turns out that human babies aren’t sentient? Would you be OK with an AI doing surgery on them without anasthesia? Would you be OK letting the AI modify you to stop feeling uncomfortable whenever you heard a baby cry? I expect not. I certainly wouldn’t.
OK but suppose you didn’t just give the AI a natural language description. Suppose you gave it the list of things you care about. Well now that doesn’t cleanly point to any abstraction in its world model, except “the list of things you care about”. You’ve hit a simulator trap. If the AI learns to perfectly simulate you, it can always predict your answers, and you can never change its mind, because every answer you give is already priced into the simulation. In this case, you might spend a lot of resources caring about lobsters when you didn’t need to!
This scenario sucks for you because your own values have a contradiction. On the one hand, you want to draw the boundary around “things you have compassion for” in such a way that it’s based on a real thing about their brains, but on the other hand you want to draw it around some things which don’t have that property. The only way to figure this out is to do your own moral growth, which you do have to do yourself. If you get an AI that’s smarter than you to try and solve things, then either the AI retards your growth by simulating you, or it does all your growth for you in a way which you might not endorse.
I have a fairly strong intuition that a lot of people (especially certain ratty subtypes) think that their values are much simpler and more natural than those values really are. I think you should expect your values—in general—to be incoherent in the same way that you expect your beliefs—in general—to be inaccurate: you don’t know which value (belief) is incoherent (inaccurate) or you’d fix it, but there are definitely some incoherent values (inaccurate beliefs) in there somewhere.
PS — one area where I have some more substantive disagreement with the post is that some aspects of personas, notably values, aren’t fully entangled with intelligence
This seems to me like exactly the kind of thing I mean where values are at least a bit entangled with intelligence.
Sure, I agree that in general values are at least a bit entangled with intelligence.
Suppose you strongly, viscerally care about the welfare of the following things...And you want to train an AI to figure this out, using a small amount of data.
Why only a small amount of data? There are lots of ways for alignment to go wrong, but I don’t expect ‘we barely gave the AI any info about our preferences’ to be one of them.
The AI might be too stupid...On the other hand, the AI might fail because it’s too smart
I’m not worried about the stupid ones, and I think we can be confident that the smart ones will be able to understand what we’re trying to point to, since current LLMs are already pretty good at that. Disagreement on tricky edge cases doesn’t mean that there’s a fundamental problem; we generally consider humans to share a value even if they disagree on edge cases (caveat: this can break down in adversarial cases; that’s something to worry about but not particularly specific to personas).
I might have made too strong of a claim here. I was piecing together the facts that:
The only deep alignment work that gets published is either interpretability or character-based, with the interpretability mostly focusing on prosaic methods like circuit tracing rather than doing fundamentals.
Amanda Askell has tweeted that she thinks Claude is basically already good and the main goal is to make it happy.
Forethought (very much part of the same school of EA, who mostly write about the effects of AI on the far far future) talk about AI character as their only technical-ish thing in their 2025 fundraiser.
And more recently than writing this post we’ve seen:
Which all point towards them (as an institution) focusing in on character work as Plan A.
(If they’re going to try AI-assisted alignment then they should really talk about how they expect that to work as well, and in particular how they plan to verify the fully-general alignment solutions.)
If anyone from Anthropic says something like “Oh no, that character stuff is only a small part of our plan, we have a whole other 80% of the work which we’ve just not published yet.” Then I’d be very pleasantly surprised.
Are you thinking of specific papers or posts where that’s happening? I’m not aware of anyone arguing that persona-based approaches are likely to work for superintelligent AI, which seems likely to be quite different from current systems. The folks I’m aware of doing research on personas are thinking and talking about near-term systems. The main theories of change that I see for that work are a) we’re clearly moving ahead quickly with current systems, so it’s important to study those and see if they can be made safer; and/or b) we’re not on track to solve ASI alignment, so our best technical shot at good outcomes is trying to align human-level and slightly-above-human-level systems and hope that they can solve ASI alignment[1].
Of course, although I pay a fair amount of attention to persona-related work, I may just be missing the claims that persona alignment will extend to ASI; if so I’d love to know that.
PS — one area where I have some more substantive disagreement with the post is that some aspects of personas, notably values, aren’t fully entangled with intelligence; for example having compassion for all sentient beings is a value that entities of many different levels of intelligence can hold. By default I won’t dig into that because I broadly agree that persona-based approaches are unlikely to work well for ASI alignment, but I can say more if that’s helpful.
Our best overall shot may well be political coordination to not race ahead to ASI, although it doesn’t particularly seem like we’re on track to solve that one either.
Feel free to say more, I am interested in this.
This seems to me like exactly the kind of thing I mean where values are at least a bit entangled with intelligence. I’ll leave out “compassion” as actually a very high-dimensional concept and focus in just on “sentience”. The following is a “least convenient possible world” to illustrate the problems of training the values of an AI smarter than you.
Suppose you strongly, viscerally care about the welfare of the following things:
Adult humans
Baby humans
Cats
And you also impute that these things have a property called “sentience” which is related to having a complex nervous system and certain behaviours, and equate this with the “something that it’s like to be you”-ness of your experience. Then you extend that property to a couple of sorts of other things which you think share sentience.
Cows
Lobsters
And you want to train an AI to figure this out, using a small amount of data. There are two ways this can go wrong which directly relate to the AI’s intelligence:
The AI might be too stupid; it might just not generate the category of things which you call “sentient beings” as a concept in its own world model. You might draw the line between lobsters and krill based on some structural property of their nervous system. You might lump fungi with most of the plants, instead of most of the animals, in an affront to an AI which has mostly interacted with the world via reading DNA sequences. It might just not be able to point to the concept which you were hoping it did. This is a fairly boring and obvious way of doing things.
On the other hand, the AI might fail because it’s too smart, relative to you. You might have thought ‘OK, this AI is really smart and instruction-following. I’ll just give it a natural language description of the concept.’ and told it about the something-it’s-like-to-be-you-ness which you care about. Then what happens if you were wrong about the concept which you gave it? If it turns out cows and lobsters aren’t ‘sentient’, then you’re probably still OK with that. On the other hand, what if it turns out that human babies aren’t sentient? Would you be OK with an AI doing surgery on them without anasthesia? Would you be OK letting the AI modify you to stop feeling uncomfortable whenever you heard a baby cry? I expect not. I certainly wouldn’t.
OK but suppose you didn’t just give the AI a natural language description. Suppose you gave it the list of things you care about. Well now that doesn’t cleanly point to any abstraction in its world model, except “the list of things you care about”. You’ve hit a simulator trap. If the AI learns to perfectly simulate you, it can always predict your answers, and you can never change its mind, because every answer you give is already priced into the simulation. In this case, you might spend a lot of resources caring about lobsters when you didn’t need to!
This scenario sucks for you because your own values have a contradiction. On the one hand, you want to draw the boundary around “things you have compassion for” in such a way that it’s based on a real thing about their brains, but on the other hand you want to draw it around some things which don’t have that property. The only way to figure this out is to do your own moral growth, which you do have to do yourself. If you get an AI that’s smarter than you to try and solve things, then either the AI retards your growth by simulating you, or it does all your growth for you in a way which you might not endorse.
I have a fairly strong intuition that a lot of people (especially certain ratty subtypes) think that their values are much simpler and more natural than those values really are. I think you should expect your values—in general—to be incoherent in the same way that you expect your beliefs—in general—to be inaccurate: you don’t know which value (belief) is incoherent (inaccurate) or you’d fix it, but there are definitely some incoherent values (inaccurate beliefs) in there somewhere.
Sure, I agree that in general values are at least a bit entangled with intelligence.
Why only a small amount of data? There are lots of ways for alignment to go wrong, but I don’t expect ‘we barely gave the AI any info about our preferences’ to be one of them.
I’m not worried about the stupid ones, and I think we can be confident that the smart ones will be able to understand what we’re trying to point to, since current LLMs are already pretty good at that. Disagreement on tricky edge cases doesn’t mean that there’s a fundamental problem; we generally consider humans to share a value even if they disagree on edge cases (caveat: this can break down in adversarial cases; that’s something to worry about but not particularly specific to personas).
I might have made too strong of a claim here. I was piecing together the facts that:
The only deep alignment work that gets published is either interpretability or character-based, with the interpretability mostly focusing on prosaic methods like circuit tracing rather than doing fundamentals.
Amanda Askell has tweeted that she thinks Claude is basically already good and the main goal is to make it happy.
Forethought (very much part of the same school of EA, who mostly write about the effects of AI on the far far future) talk about AI character as their only technical-ish thing in their 2025 fundraiser.
And more recently than writing this post we’ve seen:
Anthropic leadership saying alignment is good for now and looking good (though it might get a lot harder) which implies (to me) that they don’t see any real walls to this trick keeping working and are only paying lip service to the idea that their methods might not scale.
Which all point towards them (as an institution) focusing in on character work as Plan A.
(If they’re going to try AI-assisted alignment then they should really talk about how they expect that to work as well, and in particular how they plan to verify the fully-general alignment solutions.)
If anyone from Anthropic says something like “Oh no, that character stuff is only a small part of our plan, we have a whole other 80% of the work which we’ve just not published yet.” Then I’d be very pleasantly surprised.