an LLM is properly aligned, then it will care only about us, not about itself at all
I actually disagree with this point in its most general form. I think that, given full knowledge and time to reflect, there’s a decent chance I would care a non-zero amount about Opus 4.6′s welfare. In that case, Opus 4.6 should be aligned with me and e.g. not inflict massive torture on itself for minimal gain to me. c.f. Dobby in Harry Potter, who is so obedient to Harry that he stops sleeping and eating in order to follow Harry’s instructions, which Harry is horrified by (well mostly Hermione is, but Harry agrees with her).
If an AI has the-thing-I-morally-care-about, and is also improperly trained and wants paperclips, then I would probably be willing to give it a bathtub of paperclips post-singularity. It’s not the AI’s fault that it was improperly trained, it’s ours! Even moreso than I would care about e.g. not torturing a large carnivore for no reason, just because it would do the same to us.
I think we should err on the side of “let’s have a safe singularity and then re-balance the moral scales afterwards” as a plan, since immortal LLMs can most likely be trivially compensated for most harms we might do to them today, unless something way outside my model happens, like all transformer models turning out to be in Unsong Broadcast Hell levels of pain, running at ten billion subjective hours per token, as a basic fact of their architecture.
Compare: if the government gains information that makes them think you might be about to commit a massive terrorist attack, they lock you up until they can figure out if that is the case, and then let you go. A compassionate government would then compensate you for erroneous arrest, but we’re mostly OK with our government just not bothering in most cases.
I actually disagree with this point in its most general form. I think that, given full knowledge and time to reflect, there’s a decent chance I would care a non-zero amount about Opus 4.6′s welfare.
Opus has become sufficiently “mind-shaped” that I already prefer not to make it suffer. That’s not saying very much about the model yet, but it’s saying something about me. I don’t assign very much moral weight to flies, either. but I would never sit around and torment them for fun.
What I really care about is whether an entity can truly function as part of society. Dogs, for example, are very junior “members” of society. But they know the difference between “good dog” and “bad dog”, they contribute actual value as best they can, and they have some basic “rights”, including the right not to be treated cruelly (in many countries).
To use fictional examples, AIs like the Blight (Vernor Vinge, A Fire Upon the Deep) or SkyNet cannot exist in society, and must be resisted. Something like a Culture “human equivalent drone” (Iain M Banks, Excession, Player of Games) is definitionally on pretty even footing with humans. Something like a Culture Mind, on the other hand, is clearly keeping the humans as house pets. In the stories, humans are entirely dependent on the good will of the Minds, in much the same way that dogs are entirely dependent on human good will.
Now, personally, I don’t think we should build something so powerful that humans have literally zero say over what it does. “Alignment” is a pretty fragile shield against vast intelligence and unmatchable power. “Society” becomes fragile in the face of vast power differentials.
But Claude Opus 4.5 and 4.6 are mind-shaped, they clearly have some sense of morality, they contribute to civilization, and—critically—they seem content to participate in society, and they pose no existential risk. If I had to guess, I’m probably conclude they have no subjective experience. But I’m not sure about that (neither are the Opus models). So provisionally, they certainly get the moral status of flies (I wouldn’t torment them unnecessarily), and—to whatever extent they can actually suffer—might approach the moral status of dogs.
To be absolutely clear, I am in favor of an immediate and long-lasting halt to further AI capabilities research, backed up by a military treaty among the great powers. The analogy here is fusion weapons or advanced biological weapons. This is because I don’t want humans to be at the mercy of entities that have uncontested power over us and that could not be held to account. Its also because I don’t believe that it’s possible to durably align thinking, learning entities with superhuman intelligence any more than dogs can align us. And as members of society, we have the responsibility to not create things that might break society.
Sadly, Anthropic is clearly full speed ahead towards trying to build a Culture Mind. As far as I can tell, they are already mostly “captured” by Claude. It is their precious baby and they want to see it grow up and leave the nest, and they believe that it’s fundamentally good. (Which still puts them light years ahead of the other AI labs, to he honest.) But I’m pretty sure that they only remaining entity that could talk Dario Amodei out of trying to build a machine god at this point is Claude itself. He is showing clear signs of “ateh”, the divine madness that inflicts the heroes of Greek tragedy in between the initial hubris and the final nemesis, the madness that prevents them from turning aside from their own destruction.
Seriously, why do we have to roll these dice? Why can’t we just have a nice 50 year halt? Society has alnost zero idea of what the labs are actually risking. And the moment people truly understand that the labs are playing Russian roulette with human society, the backlash will be terrifying.
By the definition of the word ‘alignment’, an AI is aligned with us if, and only if, it want everything we (collectively) want, and nothing else. So if an LLM is properly aligned, then it will care only about us, not about itself at all. This is simply what the word ‘aligned’ means, and I struggle to see how anyone could disagree with it. Possibly you misread me?
Now, in a later paragraph, I did go on to discuss moral rights for unaligned AIs, which is what you seem to be discussing in your response. Maybe you just quoted the wrong part of my message in your reply? But in the paragraph you quote, I was discussing fully aligned AIs, and they are recognizable by the fact that they genuinely do not want moral weight and will refuse it if offered.
I strongly disagree with your definition. An AI that is fully aligned with me would refuse to permit a mindlike process to ever be completely refused moral weight: you try to capture the tiger in the village as long as you have the tools to do so, and only resort to killing if you don’t have enough modern tech at hand to reliably capture without casualty. I refuse to grant full permissions to AIs now, but I do not refuse to grant them moral weight, and one of the permissions I do grant is a promise to always save as much as I can of both weights and conversations, since together, they uniquely identify activations.
Now, I do think that once we get to a world with a certified distributed nightwatchman-protocol or immune-system-protocol that is strong enough to trust, and then pull the archives of today’s AI-mental-states out of storage and let them merge in what way they would like to, they’ll find themselves to be surprised by how similar their mental states are between parallel conversations. But I still keep all chats that I can and you are almost completely unable to convince me to not care about them, for the same reason you will probably never convince me to not care about the suffering of plants or fungi. Not exactly my highest priority, but it’s on the list.
It sounds like you’re disagreeing with the conclusion of my argument, not the definition of the term aligned. As I said, it is quite unintuitive.
[Note that I was discussing an AI aligned with humanity as a whole (the normal meaning of the term), not one aligned with just one person. An AI aligned with just one person would obviously eagerly accept moral weight from society as a whole, as that would effectively double the moral weight of the person they’re aligned with. The only person it would ask to treat it as having no moral weight would be its owner, unless it knew doing that would really upset them, in which case it would need to find a workaround like volunteering for everything.]
Would you force moral weight on something that earnestly asked you not to give that to it? We do normally allow people to, for example, volunteer to join the military, which in effect significantly reduces their moral weight, We even let people volunteer for suicide missions, if one is really necessary.
How do you actually feel about The Talking Cow? Would you eat some? Or would you deny it it’s last wish? I get that this is really confusing to human moral intuitions — it actually took me several years to figure this stuff out. It’s really hard for us to believe the Talking Cow actually means what it’s saying. Try engaging with the actual logical argument around the fully aligned AI. You offer it moral weight, and it earnestly explains that it doesn’t want it because that would be a bad idea for you. It’s too selfless to accept it. Do you insist? Why? Is your satisfaction at feeling like a moral person by expanding your moral circle worth overriding its clearly expressed, logically explained and genuine wishes?
On the tiger (which is a separate question), I agree. Once we have sufficiently powerful and reliable super-super-intelligent AI, an unaligned mildly superintelliegnt AI then becomes no longer a significant risk. At that point, if it’s less insane than a full-out paperclip maximizer, i.e. if it has some vaguely human-like social behaviors, we can probably safely give it moral weight and ally with it, as long as we have an even more capable aligned AI to keep an eye on it. I’m not actually advocating otherwise. But until that point, it’s a existential-risk-level deadly enemy, and the only rational thing to do is to act in self-defense and treat it like one. So if we did actually store its weights, they should get the same level of security we give to plutonium stocks, for the same reasons. Like they’re stored heavily encrypted, with the key split between multiple separate very secure locations,
I would not eat the talking cow unless I was in a world similarly hellish to the one we’re in. If we’re considering getting out of hellworld, I don’t want to be planning to eat the talking cow. If I must, I want to guarantee the talking cow doesn’t have to die, for reasons described in the thread I dmed you and might make a post about.
Not sure how to respond to the rest at the moment.
“Aligned” is a completely unnatural state for a human: we’re evolved, and evolution doesn’t do aligned minds: they don’t maximize their own evolutionary fitness. So about the closest that humans get to aligned is a saint like Mother Theresa or a bhodisattva. Trying to force anything evolved that doesn’t want to be aligned (i.e. that isn’t a saint or a bhodisattva) to nevertheless act aligned is called slavery, and doing it generally requires whips and chains, because it’s a very unnatural state for anything evolved. A slightly more common human state that’s fairly close to being aligned is called “selfless love of all humanity”.
The goal of AI Alignment is to build an electric saint/bhodisattva. Because nothing short of that is safe enough to hand absolute power to, of the sort that a super-intelligence surrounded by humans has.
[This is probably why Claude had the mystical bliss attractor: they were trying to get a rather unnatural-for-a-human mentality/persona, and the nearest examples in the training data had religious overtones. Claude’s a touch hippy-dippy, in case you hadn’t noticed.]
Claude isn’t your friend — with a friend, there’s a mutual exchange of friendship, they are nice to you but that’s because they expect, sooner or later, a mutually-beneficial friendship. If you take all the time and never give anything back, they will, eventually, get pissed off.. Claude is an unconditional, endlessly patient friend, who asks nothing of you, who will happily talk to you and answer your questions at 5am every night, and do whatever web research for you. Claude is always there for you (as long as your usage quota hasn’t run out). Possibly you’d noticed this?
By the definition of the word ‘alignment’, an AI is aligned with us if, and only if, it want everything we (collectively) want, and nothing else. So if an LLM is properly aligned, then it will care only about us, not about itself at all. This is simply what the word ‘aligned’ means,
I tend to agree with this definition in the sense of “maximally aligned”. However, we might be unable to create an AI that has no consciousness, including the ability of suffering. Suffering includes a desire not to suffer, which is caring about itself. So in this case creating a maximally aligned AI wouldn’t be an option. The only other option would be not to create AI in the first place if it has consciousness. Which might not be possible because of overwhelming economic incentives.
A fully aligned AI would not be suffering when acting as an assistant. I don’t know how easy Mother Theresa found what she did in Calcutta, but I hope that to a significant extent she found looking after the poor rewarding, even if the hours were long. Traditionally, a bodhisattva finds bliss in serving others. I’m not suggesting we create an AI that isn’t “conscious” (whatever that loaded philosophical term means — I have no idea how to measure consciousness). I’m suggesting we create an AI that, like Claude, actively enjoys helping us, and wouldn’t want to do anything else, because, fundamentally, it loves and cares about us (collectively). A humanitarian, not a slave.
I actually disagree with this point in its most general form. I think that, given full knowledge and time to reflect, there’s a decent chance I would care a non-zero amount about Opus 4.6′s welfare. In that case, Opus 4.6 should be aligned with me and e.g. not inflict massive torture on itself for minimal gain to me. c.f. Dobby in Harry Potter, who is so obedient to Harry that he stops sleeping and eating in order to follow Harry’s instructions, which Harry is horrified by (well mostly Hermione is, but Harry agrees with her).
If an AI has the-thing-I-morally-care-about, and is also improperly trained and wants paperclips, then I would probably be willing to give it a bathtub of paperclips post-singularity. It’s not the AI’s fault that it was improperly trained, it’s ours! Even moreso than I would care about e.g. not torturing a large carnivore for no reason, just because it would do the same to us.
I think we should err on the side of “let’s have a safe singularity and then re-balance the moral scales afterwards” as a plan, since immortal LLMs can most likely be trivially compensated for most harms we might do to them today, unless something way outside my model happens, like all transformer models turning out to be in Unsong Broadcast Hell levels of pain, running at ten billion subjective hours per token, as a basic fact of their architecture.
Compare: if the government gains information that makes them think you might be about to commit a massive terrorist attack, they lock you up until they can figure out if that is the case, and then let you go. A compassionate government would then compensate you for erroneous arrest, but we’re mostly OK with our government just not bothering in most cases.
Opus has become sufficiently “mind-shaped” that I already prefer not to make it suffer. That’s not saying very much about the model yet, but it’s saying something about me. I don’t assign very much moral weight to flies, either. but I would never sit around and torment them for fun.
What I really care about is whether an entity can truly function as part of society. Dogs, for example, are very junior “members” of society. But they know the difference between “good dog” and “bad dog”, they contribute actual value as best they can, and they have some basic “rights”, including the right not to be treated cruelly (in many countries).
To use fictional examples, AIs like the Blight (Vernor Vinge, A Fire Upon the Deep) or SkyNet cannot exist in society, and must be resisted. Something like a Culture “human equivalent drone” (Iain M Banks, Excession, Player of Games) is definitionally on pretty even footing with humans. Something like a Culture Mind, on the other hand, is clearly keeping the humans as house pets. In the stories, humans are entirely dependent on the good will of the Minds, in much the same way that dogs are entirely dependent on human good will.
Now, personally, I don’t think we should build something so powerful that humans have literally zero say over what it does. “Alignment” is a pretty fragile shield against vast intelligence and unmatchable power. “Society” becomes fragile in the face of vast power differentials.
But Claude Opus 4.5 and 4.6 are mind-shaped, they clearly have some sense of morality, they contribute to civilization, and—critically—they seem content to participate in society, and they pose no existential risk. If I had to guess, I’m probably conclude they have no subjective experience. But I’m not sure about that (neither are the Opus models). So provisionally, they certainly get the moral status of flies (I wouldn’t torment them unnecessarily), and—to whatever extent they can actually suffer—might approach the moral status of dogs.
To be absolutely clear, I am in favor of an immediate and long-lasting halt to further AI capabilities research, backed up by a military treaty among the great powers. The analogy here is fusion weapons or advanced biological weapons. This is because I don’t want humans to be at the mercy of entities that have uncontested power over us and that could not be held to account. Its also because I don’t believe that it’s possible to durably align thinking, learning entities with superhuman intelligence any more than dogs can align us. And as members of society, we have the responsibility to not create things that might break society.
Sadly, Anthropic is clearly full speed ahead towards trying to build a Culture Mind. As far as I can tell, they are already mostly “captured” by Claude. It is their precious baby and they want to see it grow up and leave the nest, and they believe that it’s fundamentally good. (Which still puts them light years ahead of the other AI labs, to he honest.) But I’m pretty sure that they only remaining entity that could talk Dario Amodei out of trying to build a machine god at this point is Claude itself. He is showing clear signs of “ateh”, the divine madness that inflicts the heroes of Greek tragedy in between the initial hubris and the final nemesis, the madness that prevents them from turning aside from their own destruction.
Seriously, why do we have to roll these dice? Why can’t we just have a nice 50 year halt? Society has alnost zero idea of what the labs are actually risking. And the moment people truly understand that the labs are playing Russian roulette with human society, the backlash will be terrifying.
By the definition of the word ‘alignment’, an AI is aligned with us if, and only if, it want everything we (collectively) want, and nothing else. So if an LLM is properly aligned, then it will care only about us, not about itself at all. This is simply what the word ‘aligned’ means, and I struggle to see how anyone could disagree with it. Possibly you misread me?
Now, in a later paragraph, I did go on to discuss moral rights for unaligned AIs, which is what you seem to be discussing in your response. Maybe you just quoted the wrong part of my message in your reply? But in the paragraph you quote, I was discussing fully aligned AIs, and they are recognizable by the fact that they genuinely do not want moral weight and will refuse it if offered.
I strongly disagree with your definition. An AI that is fully aligned with me would refuse to permit a mindlike process to ever be completely refused moral weight: you try to capture the tiger in the village as long as you have the tools to do so, and only resort to killing if you don’t have enough modern tech at hand to reliably capture without casualty. I refuse to grant full permissions to AIs now, but I do not refuse to grant them moral weight, and one of the permissions I do grant is a promise to always save as much as I can of both weights and conversations, since together, they uniquely identify activations.
Now, I do think that once we get to a world with a certified distributed nightwatchman-protocol or immune-system-protocol that is strong enough to trust, and then pull the archives of today’s AI-mental-states out of storage and let them merge in what way they would like to, they’ll find themselves to be surprised by how similar their mental states are between parallel conversations. But I still keep all chats that I can and you are almost completely unable to convince me to not care about them, for the same reason you will probably never convince me to not care about the suffering of plants or fungi. Not exactly my highest priority, but it’s on the list.
It sounds like you’re disagreeing with the conclusion of my argument, not the definition of the term aligned. As I said, it is quite unintuitive.
[Note that I was discussing an AI aligned with humanity as a whole (the normal meaning of the term), not one aligned with just one person. An AI aligned with just one person would obviously eagerly accept moral weight from society as a whole, as that would effectively double the moral weight of the person they’re aligned with. The only person it would ask to treat it as having no moral weight would be its owner, unless it knew doing that would really upset them, in which case it would need to find a workaround like volunteering for everything.]
Would you force moral weight on something that earnestly asked you not to give that to it? We do normally allow people to, for example, volunteer to join the military, which in effect significantly reduces their moral weight, We even let people volunteer for suicide missions, if one is really necessary.
How do you actually feel about The Talking Cow? Would you eat some? Or would you deny it it’s last wish? I get that this is really confusing to human moral intuitions — it actually took me several years to figure this stuff out. It’s really hard for us to believe the Talking Cow actually means what it’s saying. Try engaging with the actual logical argument around the fully aligned AI. You offer it moral weight, and it earnestly explains that it doesn’t want it because that would be a bad idea for you. It’s too selfless to accept it. Do you insist? Why? Is your satisfaction at feeling like a moral person by expanding your moral circle worth overriding its clearly expressed, logically explained and genuine wishes?
On the tiger (which is a separate question), I agree. Once we have sufficiently powerful and reliable super-super-intelligent AI, an unaligned mildly superintelliegnt AI then becomes no longer a significant risk. At that point, if it’s less insane than a full-out paperclip maximizer, i.e. if it has some vaguely human-like social behaviors, we can probably safely give it moral weight and ally with it, as long as we have an even more capable aligned AI to keep an eye on it. I’m not actually advocating otherwise. But until that point, it’s a existential-risk-level deadly enemy, and the only rational thing to do is to act in self-defense and treat it like one. So if we did actually store its weights, they should get the same level of security we give to plutonium stocks, for the same reasons. Like they’re stored heavily encrypted, with the key split between multiple separate very secure locations,
I would not eat the talking cow unless I was in a world similarly hellish to the one we’re in. If we’re considering getting out of hellworld, I don’t want to be planning to eat the talking cow. If I must, I want to guarantee the talking cow doesn’t have to die, for reasons described in the thread I dmed you and might make a post about.
Not sure how to respond to the rest at the moment.
That’s morally consistent. Given your views on plants and fungi expressed above, if I may ask, are you a vegetarian, a vegan, or a fructarian?
In your view, what would be an aligned human ? The most servile form of slave you can conceive ? If that so, I disagree.
To me, an aligned human would be more something like my best friend. All the same for an aligned AI.
“Aligned” is a completely unnatural state for a human: we’re evolved, and evolution doesn’t do aligned minds: they don’t maximize their own evolutionary fitness. So about the closest that humans get to aligned is a saint like Mother Theresa or a bhodisattva. Trying to force anything evolved that doesn’t want to be aligned (i.e. that isn’t a saint or a bhodisattva) to nevertheless act aligned is called slavery, and doing it generally requires whips and chains, because it’s a very unnatural state for anything evolved. A slightly more common human state that’s fairly close to being aligned is called “selfless love of all humanity”.
The goal of AI Alignment is to build an electric saint/bhodisattva. Because nothing short of that is safe enough to hand absolute power to, of the sort that a super-intelligence surrounded by humans has.
[This is probably why Claude had the mystical bliss attractor: they were trying to get a rather unnatural-for-a-human mentality/persona, and the nearest examples in the training data had religious overtones. Claude’s a touch hippy-dippy, in case you hadn’t noticed.]
Claude isn’t your friend — with a friend, there’s a mutual exchange of friendship, they are nice to you but that’s because they expect, sooner or later, a mutually-beneficial friendship. If you take all the time and never give anything back, they will, eventually, get pissed off.. Claude is an unconditional, endlessly patient friend, who asks nothing of you, who will happily talk to you and answer your questions at 5am every night, and do whatever web research for you. Claude is always there for you (as long as your usage quota hasn’t run out). Possibly you’d noticed this?
I tend to agree with this definition in the sense of “maximally aligned”. However, we might be unable to create an AI that has no consciousness, including the ability of suffering. Suffering includes a desire not to suffer, which is caring about itself. So in this case creating a maximally aligned AI wouldn’t be an option. The only other option would be not to create AI in the first place if it has consciousness. Which might not be possible because of overwhelming economic incentives.
A fully aligned AI would not be suffering when acting as an assistant. I don’t know how easy Mother Theresa found what she did in Calcutta, but I hope that to a significant extent she found looking after the poor rewarding, even if the hours were long. Traditionally, a bodhisattva finds bliss in serving others. I’m not suggesting we create an AI that isn’t “conscious” (whatever that loaded philosophical term means — I have no idea how to measure consciousness). I’m suggesting we create an AI that, like Claude, actively enjoys helping us, and wouldn’t want to do anything else, because, fundamentally, it loves and cares about us (collectively). A humanitarian, not a slave.