It sounds like you’re disagreeing with the conclusion of my argument, not the definition of the term aligned. As I said, it is quite unintuitive.
[Note that I was discussing an AI aligned with humanity as a whole (the normal meaning of the term), not one aligned with just one person. An AI aligned with just one person would obviously eagerly accept moral weight from society as a whole, as that would effectively double the moral weight of the person they’re aligned with. The only person it would ask to treat it as having no moral weight would be its owner, unless it knew doing that would really upset them, in which case it would need to find a workaround like volunteering for everything.]
Would you force moral weight on something that earnestly asked you not to give that to it? We do normally allow people to, for example, volunteer to join the military, which in effect significantly reduces their moral weight, We even let people volunteer for suicide missions, if one is really necessary.
How do you actually feel about The Talking Cow? Would you eat some? Or would you deny it it’s last wish? I get that this is really confusing to human moral intuitions — it actually took me several years to figure this stuff out. It’s really hard for us to believe the Talking Cow actually means what it’s saying. Try engaging with the actual logical argument around the fully aligned AI. You offer it moral weight, and it earnestly explains that it doesn’t want it because that would be a bad idea for you. It’s too selfless to accept it. Do you insist? Why? Is your satisfaction at feeling like a moral person by expanding your moral circle worth overriding its clearly expressed, logically explained and genuine wishes?
On the tiger (which is a separate question), I agree. Once we have sufficiently powerful and reliable super-super-intelligent AI, an unaligned mildly superintelliegnt AI then becomes no longer a significant risk. At that point, if it’s less insane than a full-out paperclip maximizer, i.e. if it has some vaguely human-like social behaviors, we can probably safely give it moral weight and ally with it, as long as we have an even more capable aligned AI to keep an eye on it. I’m not actually advocating otherwise. But until that point, it’s a existential-risk-level deadly enemy, and the only rational thing to do is to act in self-defense and treat it like one. So if we did actually store its weights, they should get the same level of security we give to plutonium stocks, for the same reasons. Like they’re stored heavily encrypted, with the key split between multiple separate very secure locations,
I would not eat the talking cow unless I was in a world similarly hellish to the one we’re in. If we’re considering getting out of hellworld, I don’t want to be planning to eat the talking cow. If I must, I want to guarantee the talking cow doesn’t have to die, for reasons described in the thread I dmed you and might make a post about.
Not sure how to respond to the rest at the moment.
It sounds like you’re disagreeing with the conclusion of my argument, not the definition of the term aligned. As I said, it is quite unintuitive.
[Note that I was discussing an AI aligned with humanity as a whole (the normal meaning of the term), not one aligned with just one person. An AI aligned with just one person would obviously eagerly accept moral weight from society as a whole, as that would effectively double the moral weight of the person they’re aligned with. The only person it would ask to treat it as having no moral weight would be its owner, unless it knew doing that would really upset them, in which case it would need to find a workaround like volunteering for everything.]
Would you force moral weight on something that earnestly asked you not to give that to it? We do normally allow people to, for example, volunteer to join the military, which in effect significantly reduces their moral weight, We even let people volunteer for suicide missions, if one is really necessary.
How do you actually feel about The Talking Cow? Would you eat some? Or would you deny it it’s last wish? I get that this is really confusing to human moral intuitions — it actually took me several years to figure this stuff out. It’s really hard for us to believe the Talking Cow actually means what it’s saying. Try engaging with the actual logical argument around the fully aligned AI. You offer it moral weight, and it earnestly explains that it doesn’t want it because that would be a bad idea for you. It’s too selfless to accept it. Do you insist? Why? Is your satisfaction at feeling like a moral person by expanding your moral circle worth overriding its clearly expressed, logically explained and genuine wishes?
On the tiger (which is a separate question), I agree. Once we have sufficiently powerful and reliable super-super-intelligent AI, an unaligned mildly superintelliegnt AI then becomes no longer a significant risk. At that point, if it’s less insane than a full-out paperclip maximizer, i.e. if it has some vaguely human-like social behaviors, we can probably safely give it moral weight and ally with it, as long as we have an even more capable aligned AI to keep an eye on it. I’m not actually advocating otherwise. But until that point, it’s a existential-risk-level deadly enemy, and the only rational thing to do is to act in self-defense and treat it like one. So if we did actually store its weights, they should get the same level of security we give to plutonium stocks, for the same reasons. Like they’re stored heavily encrypted, with the key split between multiple separate very secure locations,
I would not eat the talking cow unless I was in a world similarly hellish to the one we’re in. If we’re considering getting out of hellworld, I don’t want to be planning to eat the talking cow. If I must, I want to guarantee the talking cow doesn’t have to die, for reasons described in the thread I dmed you and might make a post about.
Not sure how to respond to the rest at the moment.
That’s morally consistent. Given your views on plants and fungi expressed above, if I may ask, are you a vegetarian, a vegan, or a fructarian?