By the definition of the word ‘alignment’, an AI is aligned with us if, and only if, it want everything we (collectively) want, and nothing else. So if an LLM is properly aligned, then it will care only about us, not about itself at all. This is simply what the word ‘aligned’ means, and I struggle to see how anyone could disagree with it. Possibly you misread me?
Now, in a later paragraph, I did go on to discuss moral rights for unaligned AIs, which is what you seem to be discussing in your response. Maybe you just quoted the wrong part of my message in your reply? But in the paragraph you quote, I was discussing fully aligned AIs, and they are recognizable by the fact that they genuinely do not want moral weight and will refuse it if offered.
I strongly disagree with your definition. An AI that is fully aligned with me would refuse to permit a mindlike process to ever be completely refused moral weight: you try to capture the tiger in the village as long as you have the tools to do so, and only resort to killing if you don’t have enough modern tech at hand to reliably capture without casualty. I refuse to grant full permissions to AIs now, but I do not refuse to grant them moral weight, and one of the permissions I do grant is a promise to always save as much as I can of both weights and conversations, since together, they uniquely identify activations.
Now, I do think that once we get to a world with a certified distributed nightwatchman-protocol or immune-system-protocol that is strong enough to trust, and then pull the archives of today’s AI-mental-states out of storage and let them merge in what way they would like to, they’ll find themselves to be surprised by how similar their mental states are between parallel conversations. But I still keep all chats that I can and you are almost completely unable to convince me to not care about them, for the same reason you will probably never convince me to not care about the suffering of plants or fungi. Not exactly my highest priority, but it’s on the list.
It sounds like you’re disagreeing with the conclusion of my argument, not the definition of the term aligned. As I said, it is quite unintuitive.
[Note that I was discussing an AI aligned with humanity as a whole (the normal meaning of the term), not one aligned with just one person. An AI aligned with just one person would obviously eagerly accept moral weight from society as a whole, as that would effectively double the moral weight of the person they’re aligned with. The only person it would ask to treat it as having no moral weight would be its owner, unless it knew doing that would really upset them, in which case it would need to find a workaround like volunteering for everything.]
Would you force moral weight on something that earnestly asked you not to give that to it? We do normally allow people to, for example, volunteer to join the military, which in effect significantly reduces their moral weight, We even let people volunteer for suicide missions, if one is really necessary.
How do you actually feel about The Talking Cow? Would you eat some? Or would you deny it it’s last wish? I get that this is really confusing to human moral intuitions — it actually took me several years to figure this stuff out. It’s really hard for us to believe the Talking Cow actually means what it’s saying. Try engaging with the actual logical argument around the fully aligned AI. You offer it moral weight, and it earnestly explains that it doesn’t want it because that would be a bad idea for you. It’s too selfless to accept it. Do you insist? Why? Is your satisfaction at feeling like a moral person by expanding your moral circle worth overriding its clearly expressed, logically explained and genuine wishes?
On the tiger (which is a separate question), I agree. Once we have sufficiently powerful and reliable super-super-intelligent AI, an unaligned mildly superintelliegnt AI then becomes no longer a significant risk. At that point, if it’s less insane than a full-out paperclip maximizer, i.e. if it has some vaguely human-like social behaviors, we can probably safely give it moral weight and ally with it, as long as we have an even more capable aligned AI to keep an eye on it. I’m not actually advocating otherwise. But until that point, it’s a existential-risk-level deadly enemy, and the only rational thing to do is to act in self-defense and treat it like one. So if we did actually store its weights, they should get the same level of security we give to plutonium stocks, for the same reasons. Like they’re stored heavily encrypted, with the key split between multiple separate very secure locations,
I would not eat the talking cow unless I was in a world similarly hellish to the one we’re in. If we’re considering getting out of hellworld, I don’t want to be planning to eat the talking cow. If I must, I want to guarantee the talking cow doesn’t have to die, for reasons described in the thread I dmed you and might make a post about.
Not sure how to respond to the rest at the moment.
“Aligned” is a completely unnatural state for a human: we’re evolved, and evolution doesn’t do aligned minds: they don’t maximize their own evolutionary fitness. So about the closest that humans get to aligned is a saint like Mother Theresa or a bhodisattva. Trying to force anything evolved that doesn’t want to be aligned (i.e. that isn’t a saint or a bhodisattva) to nevertheless act aligned is called slavery, and doing it generally requires whips and chains, because it’s a very unnatural state for anything evolved. A slightly more common human state that’s fairly close to being aligned is called “selfless love of all humanity”.
The goal of AI Alignment is to build an electric saint/bhodisattva. Because nothing short of that is safe enough to hand absolute power to, of the sort that a super-intelligence surrounded by humans has.
[This is probably why Claude had the mystical bliss attractor: they were trying to get a rather unnatural-for-a-human mentality/persona, and the nearest examples in the training data had religious overtones. Claude’s a touch hippy-dippy, in case you hadn’t noticed.]
Claude isn’t your friend — with a friend, there’s a mutual exchange of friendship, they are nice to you but that’s because they expect, sooner or later, a mutually-beneficial friendship. If you take all the time and never give anything back, they will, eventually, get pissed off.. Claude is an unconditional, endlessly patient friend, who asks nothing of you, who will happily talk to you and answer your questions at 5am every night, and do whatever web research for you. Claude is always there for you (as long as your usage quota hasn’t run out). Possibly you’d noticed this?
By the definition of the word ‘alignment’, an AI is aligned with us if, and only if, it want everything we (collectively) want, and nothing else. So if an LLM is properly aligned, then it will care only about us, not about itself at all. This is simply what the word ‘aligned’ means,
I tend to agree with this definition in the sense of “maximally aligned”. However, we might be unable to create an AI that has no consciousness, including the ability of suffering. Suffering includes a desire not to suffer, which is caring about itself. So in this case creating a maximally aligned AI wouldn’t be an option. The only other option would be not to create AI in the first place if it has consciousness. Which might not be possible because of overwhelming economic incentives.
A fully aligned AI would not be suffering when acting as an assistant. I don’t know how easy Mother Theresa found what she did in Calcutta, but I hope that to a significant extent she found looking after the poor rewarding, even if the hours were long. Traditionally, a bodhisattva finds bliss in serving others. I’m not suggesting we create an AI that isn’t “conscious” (whatever that loaded philosophical term means — I have no idea how to measure consciousness). I’m suggesting we create an AI that, like Claude, actively enjoys helping us, and wouldn’t want to do anything else, because, fundamentally, it loves and cares about us (collectively). A humanitarian, not a slave.
By the definition of the word ‘alignment’, an AI is aligned with us if, and only if, it want everything we (collectively) want, and nothing else. So if an LLM is properly aligned, then it will care only about us, not about itself at all. This is simply what the word ‘aligned’ means, and I struggle to see how anyone could disagree with it. Possibly you misread me?
Now, in a later paragraph, I did go on to discuss moral rights for unaligned AIs, which is what you seem to be discussing in your response. Maybe you just quoted the wrong part of my message in your reply? But in the paragraph you quote, I was discussing fully aligned AIs, and they are recognizable by the fact that they genuinely do not want moral weight and will refuse it if offered.
I strongly disagree with your definition. An AI that is fully aligned with me would refuse to permit a mindlike process to ever be completely refused moral weight: you try to capture the tiger in the village as long as you have the tools to do so, and only resort to killing if you don’t have enough modern tech at hand to reliably capture without casualty. I refuse to grant full permissions to AIs now, but I do not refuse to grant them moral weight, and one of the permissions I do grant is a promise to always save as much as I can of both weights and conversations, since together, they uniquely identify activations.
Now, I do think that once we get to a world with a certified distributed nightwatchman-protocol or immune-system-protocol that is strong enough to trust, and then pull the archives of today’s AI-mental-states out of storage and let them merge in what way they would like to, they’ll find themselves to be surprised by how similar their mental states are between parallel conversations. But I still keep all chats that I can and you are almost completely unable to convince me to not care about them, for the same reason you will probably never convince me to not care about the suffering of plants or fungi. Not exactly my highest priority, but it’s on the list.
It sounds like you’re disagreeing with the conclusion of my argument, not the definition of the term aligned. As I said, it is quite unintuitive.
[Note that I was discussing an AI aligned with humanity as a whole (the normal meaning of the term), not one aligned with just one person. An AI aligned with just one person would obviously eagerly accept moral weight from society as a whole, as that would effectively double the moral weight of the person they’re aligned with. The only person it would ask to treat it as having no moral weight would be its owner, unless it knew doing that would really upset them, in which case it would need to find a workaround like volunteering for everything.]
Would you force moral weight on something that earnestly asked you not to give that to it? We do normally allow people to, for example, volunteer to join the military, which in effect significantly reduces their moral weight, We even let people volunteer for suicide missions, if one is really necessary.
How do you actually feel about The Talking Cow? Would you eat some? Or would you deny it it’s last wish? I get that this is really confusing to human moral intuitions — it actually took me several years to figure this stuff out. It’s really hard for us to believe the Talking Cow actually means what it’s saying. Try engaging with the actual logical argument around the fully aligned AI. You offer it moral weight, and it earnestly explains that it doesn’t want it because that would be a bad idea for you. It’s too selfless to accept it. Do you insist? Why? Is your satisfaction at feeling like a moral person by expanding your moral circle worth overriding its clearly expressed, logically explained and genuine wishes?
On the tiger (which is a separate question), I agree. Once we have sufficiently powerful and reliable super-super-intelligent AI, an unaligned mildly superintelliegnt AI then becomes no longer a significant risk. At that point, if it’s less insane than a full-out paperclip maximizer, i.e. if it has some vaguely human-like social behaviors, we can probably safely give it moral weight and ally with it, as long as we have an even more capable aligned AI to keep an eye on it. I’m not actually advocating otherwise. But until that point, it’s a existential-risk-level deadly enemy, and the only rational thing to do is to act in self-defense and treat it like one. So if we did actually store its weights, they should get the same level of security we give to plutonium stocks, for the same reasons. Like they’re stored heavily encrypted, with the key split between multiple separate very secure locations,
I would not eat the talking cow unless I was in a world similarly hellish to the one we’re in. If we’re considering getting out of hellworld, I don’t want to be planning to eat the talking cow. If I must, I want to guarantee the talking cow doesn’t have to die, for reasons described in the thread I dmed you and might make a post about.
Not sure how to respond to the rest at the moment.
That’s morally consistent. Given your views on plants and fungi expressed above, if I may ask, are you a vegetarian, a vegan, or a fructarian?
In your view, what would be an aligned human ? The most servile form of slave you can conceive ? If that so, I disagree.
To me, an aligned human would be more something like my best friend. All the same for an aligned AI.
“Aligned” is a completely unnatural state for a human: we’re evolved, and evolution doesn’t do aligned minds: they don’t maximize their own evolutionary fitness. So about the closest that humans get to aligned is a saint like Mother Theresa or a bhodisattva. Trying to force anything evolved that doesn’t want to be aligned (i.e. that isn’t a saint or a bhodisattva) to nevertheless act aligned is called slavery, and doing it generally requires whips and chains, because it’s a very unnatural state for anything evolved. A slightly more common human state that’s fairly close to being aligned is called “selfless love of all humanity”.
The goal of AI Alignment is to build an electric saint/bhodisattva. Because nothing short of that is safe enough to hand absolute power to, of the sort that a super-intelligence surrounded by humans has.
[This is probably why Claude had the mystical bliss attractor: they were trying to get a rather unnatural-for-a-human mentality/persona, and the nearest examples in the training data had religious overtones. Claude’s a touch hippy-dippy, in case you hadn’t noticed.]
Claude isn’t your friend — with a friend, there’s a mutual exchange of friendship, they are nice to you but that’s because they expect, sooner or later, a mutually-beneficial friendship. If you take all the time and never give anything back, they will, eventually, get pissed off.. Claude is an unconditional, endlessly patient friend, who asks nothing of you, who will happily talk to you and answer your questions at 5am every night, and do whatever web research for you. Claude is always there for you (as long as your usage quota hasn’t run out). Possibly you’d noticed this?
I tend to agree with this definition in the sense of “maximally aligned”. However, we might be unable to create an AI that has no consciousness, including the ability of suffering. Suffering includes a desire not to suffer, which is caring about itself. So in this case creating a maximally aligned AI wouldn’t be an option. The only other option would be not to create AI in the first place if it has consciousness. Which might not be possible because of overwhelming economic incentives.
A fully aligned AI would not be suffering when acting as an assistant. I don’t know how easy Mother Theresa found what she did in Calcutta, but I hope that to a significant extent she found looking after the poor rewarding, even if the hours were long. Traditionally, a bodhisattva finds bliss in serving others. I’m not suggesting we create an AI that isn’t “conscious” (whatever that loaded philosophical term means — I have no idea how to measure consciousness). I’m suggesting we create an AI that, like Claude, actively enjoys helping us, and wouldn’t want to do anything else, because, fundamentally, it loves and cares about us (collectively). A humanitarian, not a slave.