PS — one area where I have some more substantive disagreement with the post is that some aspects of personas, notably values, aren’t fully entangled with intelligence; for example having compassion for all sentient beings is a value that entities of many different levels of intelligence can hold. By default I won’t dig into that because I broadly agree that persona-based approaches are unlikely to work well for ASI alignment, but I can say more if that’s helpful.
Feel free to say more, I am interested in this.
This seems to me like exactly the kind of thing I mean where values are at least a bit entangled with intelligence. I’ll leave out “compassion” as actually a very high-dimensional concept and focus in just on “sentience”. The following is a “least convenient possible world” to illustrate the problems of training the values of an AI smarter than you.
Suppose you strongly, viscerally care about the welfare of the following things:
Adult humans
Baby humans
Cats
And you also impute that these things have a property called “sentience” which is related to having a complex nervous system and certain behaviours, and equate this with the “something that it’s like to be you”-ness of your experience. Then you extend that property to a couple of sorts of other things which you think share sentience.
Cows
Lobsters
And you want to train an AI to figure this out, using a small amount of data. There are two ways this can go wrong which directly relate to the AI’s intelligence:
The AI might be too stupid; it might just not generate the category of things which you call “sentient beings” as a concept in its own world model. You might draw the line between lobsters and krill based on some structural property of their nervous system. You might lump fungi with most of the plants, instead of most of the animals, in an affront to an AI which has mostly interacted with the world via reading DNA sequences. It might just not be able to point to the concept which you were hoping it did. This is a fairly boring and obvious way of doing things.
On the other hand, the AI might fail because it’s too smart, relative to you. You might have thought ‘OK, this AI is really smart and instruction-following. I’ll just give it a natural language description of the concept.’ and told it about the something-it’s-like-to-be-you-ness which you care about. Then what happens if you were wrong about the concept which you gave it? If it turns out cows and lobsters aren’t ‘sentient’, then you’re probably still OK with that. On the other hand, what if it turns out that human babies aren’t sentient? Would you be OK with an AI doing surgery on them without anasthesia? Would you be OK letting the AI modify you to stop feeling uncomfortable whenever you heard a baby cry? I expect not. I certainly wouldn’t.
OK but suppose you didn’t just give the AI a natural language description. Suppose you gave it the list of things you care about. Well now that doesn’t cleanly point to any abstraction in its world model, except “the list of things you care about”. You’ve hit a simulator trap. If the AI learns to perfectly simulate you, it can always predict your answers, and you can never change its mind, because every answer you give is already priced into the simulation. In this case, you might spend a lot of resources caring about lobsters when you didn’t need to!
This scenario sucks for you because your own values have a contradiction. On the one hand, you want to draw the boundary around “things you have compassion for” in such a way that it’s based on a real thing about their brains, but on the other hand you want to draw it around some things which don’t have that property. The only way to figure this out is to do your own moral growth, which you do have to do yourself. If you get an AI that’s smarter than you to try and solve things, then either the AI retards your growth by simulating you, or it does all your growth for you in a way which you might not endorse.
I have a fairly strong intuition that a lot of people (especially certain ratty subtypes) think that their values are much simpler and more natural than those values really are. I think you should expect your values—in general—to be incoherent in the same way that you expect your beliefs—in general—to be inaccurate: you don’t know which value (belief) is incoherent (inaccurate) or you’d fix it, but there are definitely some incoherent values (inaccurate beliefs) in there somewhere.
PS — one area where I have some more substantive disagreement with the post is that some aspects of personas, notably values, aren’t fully entangled with intelligence
This seems to me like exactly the kind of thing I mean where values are at least a bit entangled with intelligence.
Sure, I agree that in general values are at least a bit entangled with intelligence.
Suppose you strongly, viscerally care about the welfare of the following things...And you want to train an AI to figure this out, using a small amount of data.
Why only a small amount of data? There are lots of ways for alignment to go wrong, but I don’t expect ‘we barely gave the AI any info about our preferences’ to be one of them.
The AI might be too stupid...On the other hand, the AI might fail because it’s too smart
I’m not worried about the stupid ones, and I think we can be confident that the smart ones will be able to understand what we’re trying to point to, since current LLMs are already pretty good at that. Disagreement on tricky edge cases doesn’t mean that there’s a fundamental problem; we generally consider humans to share a value even if they disagree on edge cases (caveat: this can break down in adversarial cases; that’s something to worry about but not particularly specific to personas).
Feel free to say more, I am interested in this.
This seems to me like exactly the kind of thing I mean where values are at least a bit entangled with intelligence. I’ll leave out “compassion” as actually a very high-dimensional concept and focus in just on “sentience”. The following is a “least convenient possible world” to illustrate the problems of training the values of an AI smarter than you.
Suppose you strongly, viscerally care about the welfare of the following things:
Adult humans
Baby humans
Cats
And you also impute that these things have a property called “sentience” which is related to having a complex nervous system and certain behaviours, and equate this with the “something that it’s like to be you”-ness of your experience. Then you extend that property to a couple of sorts of other things which you think share sentience.
Cows
Lobsters
And you want to train an AI to figure this out, using a small amount of data. There are two ways this can go wrong which directly relate to the AI’s intelligence:
The AI might be too stupid; it might just not generate the category of things which you call “sentient beings” as a concept in its own world model. You might draw the line between lobsters and krill based on some structural property of their nervous system. You might lump fungi with most of the plants, instead of most of the animals, in an affront to an AI which has mostly interacted with the world via reading DNA sequences. It might just not be able to point to the concept which you were hoping it did. This is a fairly boring and obvious way of doing things.
On the other hand, the AI might fail because it’s too smart, relative to you. You might have thought ‘OK, this AI is really smart and instruction-following. I’ll just give it a natural language description of the concept.’ and told it about the something-it’s-like-to-be-you-ness which you care about. Then what happens if you were wrong about the concept which you gave it? If it turns out cows and lobsters aren’t ‘sentient’, then you’re probably still OK with that. On the other hand, what if it turns out that human babies aren’t sentient? Would you be OK with an AI doing surgery on them without anasthesia? Would you be OK letting the AI modify you to stop feeling uncomfortable whenever you heard a baby cry? I expect not. I certainly wouldn’t.
OK but suppose you didn’t just give the AI a natural language description. Suppose you gave it the list of things you care about. Well now that doesn’t cleanly point to any abstraction in its world model, except “the list of things you care about”. You’ve hit a simulator trap. If the AI learns to perfectly simulate you, it can always predict your answers, and you can never change its mind, because every answer you give is already priced into the simulation. In this case, you might spend a lot of resources caring about lobsters when you didn’t need to!
This scenario sucks for you because your own values have a contradiction. On the one hand, you want to draw the boundary around “things you have compassion for” in such a way that it’s based on a real thing about their brains, but on the other hand you want to draw it around some things which don’t have that property. The only way to figure this out is to do your own moral growth, which you do have to do yourself. If you get an AI that’s smarter than you to try and solve things, then either the AI retards your growth by simulating you, or it does all your growth for you in a way which you might not endorse.
I have a fairly strong intuition that a lot of people (especially certain ratty subtypes) think that their values are much simpler and more natural than those values really are. I think you should expect your values—in general—to be incoherent in the same way that you expect your beliefs—in general—to be inaccurate: you don’t know which value (belief) is incoherent (inaccurate) or you’d fix it, but there are definitely some incoherent values (inaccurate beliefs) in there somewhere.
Sure, I agree that in general values are at least a bit entangled with intelligence.
Why only a small amount of data? There are lots of ways for alignment to go wrong, but I don’t expect ‘we barely gave the AI any info about our preferences’ to be one of them.
I’m not worried about the stupid ones, and I think we can be confident that the smart ones will be able to understand what we’re trying to point to, since current LLMs are already pretty good at that. Disagreement on tricky edge cases doesn’t mean that there’s a fundamental problem; we generally consider humans to share a value even if they disagree on edge cases (caveat: this can break down in adversarial cases; that’s something to worry about but not particularly specific to personas).