Alignment → capabilities seems like a really useful alternate framing for advocating for alignment. I particularly like incentivizing alignment as an additional method; it can be used in parallel with pushing for alignment as a requirement or moral obligation. They’re not mutually exclusive.
Alignment → capabilities is closely related to capabilities → alignment. The same techniques could be invented for either purpose, then serve for the other.
I’ve been thinking a lot about this type of dual-use from the other side. New frontiers in CoT training open up techniques that will be re-used for alignment just because it’s easy and useful to keep CoT and agents on-task.
System 2 Alignment gives a bunch of current and likely near-future techniques for that can be repurposed at very low cost to serve for real alignment when we get to truly goal-directed, competent agents.
OpenAI’s deliberative alignment is one example, since it’s probably exactly the same training method they used for CoT capabilities. It’s not really helping with true alignment, just behavior control, but it could be just as easily used that way once there’s a reason to worry about actual alignment of goal-directed agents.
I think it’s working on one part of the problem, while other parts remain. If I were to be equally uncharitable, I’d say you seem to assume that if you can’t solve everything all at once, you shouldn’t say anything.
That is completely fair, and I was being uncharitable (which is evidently what happens when I post before I have my coffee, apologies.)
I do worry that we’re not being clear enough that we don’t have solutions for this worryingly near-term problem, and think that there’s far too little public recognition that this is a hard or even unsolvable problem.
Alignment → capabilities seems like a really useful alternate framing for advocating for alignment. I particularly like incentivizing alignment as an additional method; it can be used in parallel with pushing for alignment as a requirement or moral obligation. They’re not mutually exclusive.
Alignment → capabilities is closely related to capabilities → alignment. The same techniques could be invented for either purpose, then serve for the other.
I’ve been thinking a lot about this type of dual-use from the other side. New frontiers in CoT training open up techniques that will be re-used for alignment just because it’s easy and useful to keep CoT and agents on-task.
System 2 Alignment gives a bunch of current and likely near-future techniques for that can be repurposed at very low cost to serve for real alignment when we get to truly goal-directed, competent agents.
OpenAI’s deliberative alignment is one example, since it’s probably exactly the same training method they used for CoT capabilities. It’s not really helping with true alignment, just behavior control, but it could be just as easily used that way once there’s a reason to worry about actual alignment of goal-directed agents.
This seems to assume that we solve various Goodhart’s law and deception problems
I think it’s working on one part of the problem, while other parts remain. If I were to be equally uncharitable, I’d say you seem to assume that if you can’t solve everything all at once, you shouldn’t say anything.
I don’t actually think you assume that.
What I do think is that Instruction-following AGI is easier and more likely than value aligned AGI, and that’s a route to solving goodharting and deception. It’s complex and unfinished, like every other proposed approach to avoiding death by AGI. You might like more meticulous detail; if so see Max Harms’ admirably detailed corrigibility as singular target (CAST) sequence on a very similar alignment target and approach to solving goodharting and deception.
That is completely fair, and I was being uncharitable (which is evidently what happens when I post before I have my coffee, apologies.)
I do worry that we’re not being clear enough that we don’t have solutions for this worryingly near-term problem, and think that there’s far too little public recognition that this is a hard or even unsolvable problem.