My version of what’s happening in this conversation is that you and Paul are like “Well, what if it wants things but in a way which is transparent/interpretable and hence controllable by humans, e.g. if it wants what it is prompted to want?” My response is “Indeed that would be super safe, but it would still count as wanting things. Nate’s post is titled “ability to solve long-horizon tasks correlates with wanting” not “ability to solve long-horizon tasks correlates with hidden uncontrollable wanting.”
One thing at time. First we establish that ability to solve long-horizon tasks correlates with wanting, then we argue about whether or not the future systems that are able to solve diverse long-horizon tasks better than humans can will have transparent controllable wants or not. As you yourself pointed out, insofar as we are doing lots of RL it’s dubious that the wants will remain as transparent and controllable as they are now. I meanwhile will agree that a large part of my hope for a technical solution comes from something like the Faithful CoT agenda, in which we build powerful agentic systems whose wants (and more generally, thoughts) are transparent and controllable.
If this is what’s going on, then I basically can’t imagine any context in which I would want someone to read the OP rather a post than showing examples of LM agents achieving goals and saying “it’s already the case that LM agents want things, more and more deployments of LMs will be agents, and those agents will become more competent such that it would be increasingly scary if they wanted something at cross-purposes to humans.” Is there something I’m missing?
I think your interpretation of Nate is probably wrong, but I’m not sure and happy to drop it.
FWIW, your proposed pitch “it’s already the case that...” is almost exactly the elevator pitch I currently go around giving. So maybe we agree? I’m not here to defend Nate’s choice to write this post rather than some other post.
Thanks for the explanation btw.
My version of what’s happening in this conversation is that you and Paul are like “Well, what if it wants things but in a way which is transparent/interpretable and hence controllable by humans, e.g. if it wants what it is prompted to want?” My response is “Indeed that would be super safe, but it would still count as wanting things. Nate’s post is titled “ability to solve long-horizon tasks correlates with wanting” not “ability to solve long-horizon tasks correlates with hidden uncontrollable wanting.”
One thing at time. First we establish that ability to solve long-horizon tasks correlates with wanting, then we argue about whether or not the future systems that are able to solve diverse long-horizon tasks better than humans can will have transparent controllable wants or not. As you yourself pointed out, insofar as we are doing lots of RL it’s dubious that the wants will remain as transparent and controllable as they are now. I meanwhile will agree that a large part of my hope for a technical solution comes from something like the Faithful CoT agenda, in which we build powerful agentic systems whose wants (and more generally, thoughts) are transparent and controllable.
If this is what’s going on, then I basically can’t imagine any context in which I would want someone to read the OP rather a post than showing examples of LM agents achieving goals and saying “it’s already the case that LM agents want things, more and more deployments of LMs will be agents, and those agents will become more competent such that it would be increasingly scary if they wanted something at cross-purposes to humans.” Is there something I’m missing?
I think your interpretation of Nate is probably wrong, but I’m not sure and happy to drop it.
FWIW, your proposed pitch “it’s already the case that...” is almost exactly the elevator pitch I currently go around giving. So maybe we agree? I’m not here to defend Nate’s choice to write this post rather than some other post.