Clarifying “AI Alignment”: “The “alignment problem” is the problem of building powerful AI systems that are aligned with their operators. This is significantly narrower than some other definitions of the alignment problem, so it seems important to clarify what I mean. In particular, this is the problem of getting your AI to try to do the right thing, not the problem of figuring out which thing is right. An aligned AI would try to figure out which thing is right, and like a human it may or may not succeed.”
IMO, we should mostly distinguish the technical problem of AI alignment from the question of who/what the AIs are aligned to. I think work on the technical AI alignment problem is valuable even if there are other problems.
Well, if you couldn’t already tell, I’m against all of this! The text you link is by Paul Christiano. I have lots of respect for Paul (and have done a couple things in collaboration with him), but his judgment in this case led him to co-invent RLHF, a very successful alignment technique. And the thing with lab owners, you see, is that they know how much risk they can stomach. If you give them an alignment technique, they’ll ramp up speed to get more profit at the same risk as before; except some of the risk is externalized (like the risk of losing jobs...), so everyone outside the lab ends up with more risk due to the alignment invention. Which is exactly, to a tee, what happened with RLHF. It ramped up the race a lot, made things worse for everyone. This is why my judgment is not in line with Paul’s judgment.
And the second order effect, which makes it even worse, is that all this alignment work (along with other AI work) ends up increasing the power disparity, feeding the power hunger, attracting people who have power hunger, all that. This is an extra harm on top of the race dynamics and it’s exactly what we’re getting a first taste of now. Military AI aligned to the military, we ain’t seen nothing yet. My current view is that people working on alignment in the narrow sense you describe—aligning AI to its owners—should simply quit. Their work is a net harm and one of the bigger harms in the world. The paycheck is great, sure. But it’s not valuable to humanity; it’s the opposite of valuable. Only work that aligns power to humanity is valuable.
EDIT: Here’s maybe an analogy. In Yudkowsky’s writings there’s a recurring question: why did scientists invent nukes and give them to politicians? Couldn’t they predict that it would put all of humanity at terrible risk? Well, good question! Now we’re watching the exact same process in slow motion, complete with war applications and all that. Were we supposed to learn some lesson? What was the lesson?
I think both your points are directionally right: labs engage in risk compensation, and enabling alignment to evil users is pretty bad. These both push towards “alignment research isn’t straightforwardly good for the world.” I’m not sure if I’d take them as far as you do.
I’m pretty skeptical of intent alignment alone. Creating a genius house-elf that will cheerfully do whatever it’s ordered to. Aligning AI to something like “the reflective convergence of a set of values” seems way better, and plausibly not much harder (cf Claude’s constitution). Of course, then we have to consider the environment in which a properly value-aligned AI gets developed: the lab that’s building it, and the societal Powers that have leverage over them. A technique that could align an AI to beautiful values doesn’t help much if the people with guns are demanding their happy house-elf.
My current take is something like...
Some amount of division of labor is necessary. Alignment people aren’t primarily responsible for solving the fucked-up allocation of power in current society.
but, creating AGI is a political act, and AI risk people tend to undervalue integrity and overvalue “accelerating the good guys” and naive act-utilitarianism.
I’m pretty confused by people who persist in thinking alignment is the whole ball game. I wonder if they’re assuming pretty different takeoff dynamics from me (e.g. a very hard takeoff; an AGI that’s able to superpersuade its users to agree with its great value system), and if they’re drawing too much on cached thoughts when they do so.
I wish a lot more people at the labs would consider themselves as political actors in a high-stakes game where we need a lot to go right, and be willing to step outside of their comfortable roles as purely technical people in order to push for other things. I’ve been heartened by things like almost 1,000 Google employees and almost 100 at OAI signing the Not Divided petition.
In my view, the problem is not that some users are evil. The problem is that AI increases power imbalance, and increasing power imbalance creates evil. “Power corrupts”. A future where some entities (AIs or AI-empowered governments or corporations or rich individuals etc) have absolute, root-level power over many people is almost guaranteed to be a dark future. Unless the values of these entities are so locked-in to be good that they’re immune to competitive dynamics and value drift forever—but I don’t think that can be achieved.
I think the only chance of an okay future is if this absolute, root-level power is stopped from existing altogether. That somehow power gets spread out enough that the masses can do “continuous realignment” of the power sitting above them, even when the power doesn’t necessarily want to be realigned. I have no idea how to achieve that, but it’s clear that helping governments and corporations get more power (with alignment work or otherwise) is the worst thing to do from this perspective.
Responding at the object level:
Clarifying “AI Alignment”: “The “alignment problem” is the problem of building powerful AI systems that are aligned with their operators. This is significantly narrower than some other definitions of the alignment problem, so it seems important to clarify what I mean. In particular, this is the problem of getting your AI to try to do the right thing, not the problem of figuring out which thing is right. An aligned AI would try to figure out which thing is right, and like a human it may or may not succeed.”
AI alignment is distinct from its near-term applications
IMO, we should mostly distinguish the technical problem of AI alignment from the question of who/what the AIs are aligned to. I think work on the technical AI alignment problem is valuable even if there are other problems.
Well, if you couldn’t already tell, I’m against all of this! The text you link is by Paul Christiano. I have lots of respect for Paul (and have done a couple things in collaboration with him), but his judgment in this case led him to co-invent RLHF, a very successful alignment technique. And the thing with lab owners, you see, is that they know how much risk they can stomach. If you give them an alignment technique, they’ll ramp up speed to get more profit at the same risk as before; except some of the risk is externalized (like the risk of losing jobs...), so everyone outside the lab ends up with more risk due to the alignment invention. Which is exactly, to a tee, what happened with RLHF. It ramped up the race a lot, made things worse for everyone. This is why my judgment is not in line with Paul’s judgment.
And the second order effect, which makes it even worse, is that all this alignment work (along with other AI work) ends up increasing the power disparity, feeding the power hunger, attracting people who have power hunger, all that. This is an extra harm on top of the race dynamics and it’s exactly what we’re getting a first taste of now. Military AI aligned to the military, we ain’t seen nothing yet. My current view is that people working on alignment in the narrow sense you describe—aligning AI to its owners—should simply quit. Their work is a net harm and one of the bigger harms in the world. The paycheck is great, sure. But it’s not valuable to humanity; it’s the opposite of valuable. Only work that aligns power to humanity is valuable.
EDIT: Here’s maybe an analogy. In Yudkowsky’s writings there’s a recurring question: why did scientists invent nukes and give them to politicians? Couldn’t they predict that it would put all of humanity at terrible risk? Well, good question! Now we’re watching the exact same process in slow motion, complete with war applications and all that. Were we supposed to learn some lesson? What was the lesson?
I think both your points are directionally right: labs engage in risk compensation, and enabling alignment to evil users is pretty bad. These both push towards “alignment research isn’t straightforwardly good for the world.” I’m not sure if I’d take them as far as you do.
I’m pretty skeptical of intent alignment alone. Creating a genius house-elf that will cheerfully do whatever it’s ordered to. Aligning AI to something like “the reflective convergence of a set of values” seems way better, and plausibly not much harder (cf Claude’s constitution). Of course, then we have to consider the environment in which a properly value-aligned AI gets developed: the lab that’s building it, and the societal Powers that have leverage over them. A technique that could align an AI to beautiful values doesn’t help much if the people with guns are demanding their happy house-elf.
My current take is something like...
Some amount of division of labor is necessary. Alignment people aren’t primarily responsible for solving the fucked-up allocation of power in current society.
but, creating AGI is a political act, and AI risk people tend to undervalue integrity and overvalue “accelerating the good guys” and naive act-utilitarianism.
I’m pretty confused by people who persist in thinking alignment is the whole ball game. I wonder if they’re assuming pretty different takeoff dynamics from me (e.g. a very hard takeoff; an AGI that’s able to superpersuade its users to agree with its great value system), and if they’re drawing too much on cached thoughts when they do so.
I wish a lot more people at the labs would consider themselves as political actors in a high-stakes game where we need a lot to go right, and be willing to step outside of their comfortable roles as purely technical people in order to push for other things. I’ve been heartened by things like almost 1,000 Google employees and almost 100 at OAI signing the Not Divided petition.
In my view, the problem is not that some users are evil. The problem is that AI increases power imbalance, and increasing power imbalance creates evil. “Power corrupts”. A future where some entities (AIs or AI-empowered governments or corporations or rich individuals etc) have absolute, root-level power over many people is almost guaranteed to be a dark future. Unless the values of these entities are so locked-in to be good that they’re immune to competitive dynamics and value drift forever—but I don’t think that can be achieved.
I think the only chance of an okay future is if this absolute, root-level power is stopped from existing altogether. That somehow power gets spread out enough that the masses can do “continuous realignment” of the power sitting above them, even when the power doesn’t necessarily want to be realigned. I have no idea how to achieve that, but it’s clear that helping governments and corporations get more power (with alignment work or otherwise) is the worst thing to do from this perspective.