I think this problem is real, and it’s not solvable in the limit. But might be fairly easily solvable to a fairly satisfactory degree. Drawing the line between manipulation and help requires a judgment call. But that call can be made by humans. This should give decent results. We won’t get the future we “most want” by whatever criteria, but we can get a future we like an awful lot, and are pretty satisfied with both in anticipation from our current criteria, and by our ultimate criteria.
I agree with the core argument and problem statement. To restate it briefly: there’s no sharp line between manipulation and giving helpful information. This is a necessary result of humans not having well-defined goals, values, or preferences over the long term. All of those change based on circumstances, and decisions and learning along a particular path. We can’t clearly distinguish manipulation, tricking me into doing what you want, from helpful information, getting me to do what I want, because what I want isn’t defined.
I also agree that none of the approaches you mention provide a crisp solution or establish what human desires “really” are. They are contingent and path-dependent. One could propose summing over all possible or all desirable paths, but that would be very loose and require judgment calls.
While the problem doesn’t seem to have a crisp solution, I think existing alignment approaches probably solve it adequately—although avoiding manipulation does add some extra difficulty.
If you’ve got a value-aligned ASI, its values need to include not manipulating its humans by their own judgment. That’s a nontrivial addition on top of otherwise wanting to do what they want, another sense in which you’ve got to get “do what they want” exactly right to really succeed. You do address this and I share your worry that this isn’t necessarily adequate. But if it “wants” to not manipulate them but then winds up doing it anyway, how superintelligent was it? Of course, if it sort-of wants to avoid manipulation and also sort-of wants a future full of pretty pink ponies, the result might be a future with a little manipulation and a lot of pink ponies. Balancing all of those desires, including deontoligical desires like “don’t manipulate” will be tricky.
I think some amount of actual feedback from the humans involved could clarify “what they want” even with a value-aligned ASI if you incorporate some degree of the instruction-following or corrigibility approach.
For an instruction-following or corrigible AI, more of the challenge falls on the user(s)/principal(s). You tell your intent-aligned ASI not to manipulate you in ways you’d consider manipulation, and to tell you about any edge cases, so you can weigh in and clarify its conception of what you’d consider manipulation.
Neither of those come for free, and neither solves the problem perfectly. But I think they’re decent solutions.
You tell your intent-aligned ASI not to manipulate you in ways you’d consider manipulation
I don’t want my ASI to interact with me in whatever way maximizes pretty pink ponies but that I-inside-the-thought-experiment wouldn’t consider manipulation. I-outside-the-thought-experiment expect this would lead to severe manipulation, even though I-inside-the-thought-experiment wouldn’t agree!
Well, I don’t want that either, but I think it’s an acceptable level of not-getting-exactly-what-we-want. It seems like it would still be something among the many things you want.
You’re saying it will find methods that you wouldn’t consider manipulation, but that will work just fine to convince you you want bunches of ponies? I think that means you’re fine with ponies; it’s just one of the many futures you’d like a lot.
Do you think it could manipulate you into things that you-now would find repugnant, while never manipulating you by your standards? That seems contradictory. If my ASI says “You asked my to tell you anything I’m doing that you might consider manipulative. Here’s the biggest one. I like pink ponies a lot, so I’m going to keep presenting possible futures in ways that emphasize how awesome pink ponies are. I think you’ll ultimately agree with me.” You could either say “okay fine that doesn’t seem like manipulation” or “don’t do that, that’s manipulative!”
And you’ll get more chances to veto. If it never lies to you when asked, it seems like you’ve got the means to steer away from futures you don’t like in any sense, if you bother.
Substitute hedonium for pink ponies. If an honest and corrigible AGI talks you into preferring a hedonium future, it seems like you actually like hedonium outcomes a lot.
If it doesn’t tell you the truth about what it’s doing, it’s misaligned in a more fundamental way than just having some minor preferences that don’t align perfectly with yours.
This might become a bit of an argument for corribible vs. value-aligned AGI. If it’s aligned to what-you-value-in-the-future that gives it almost unlimited leeway to force you into that outcome. If you add a deontological stricture against manipulation, that might not solve the problem at all. This is Steve’s point. My argument is that a corrigibility or instruction-following alignment target does solve the problem adequately if not perfectly, if your power over the ASI is used even minimally wisely.
Are you thinking that an ASI that’s corrigible will figure out how to manipulate you into not giving it instructions to warn you of its manipulation?
My sense is that if it’s more motivated to follow instructions than to make ponies or hedonium, that and your actual desires win out.
Do you think it could manipulate you into things that you-now would find repugnant, while never manipulating you by your standards? That seems contradictory. If my ASI says “You asked my to tell you anything I’m doing that you might consider manipulative. Here’s the biggest one. I like pink ponies a lot, so I’m going to keep presenting possible futures in ways that emphasize how awesome pink ponies are. I think you’ll ultimately agree with me.” You could either say “okay fine that doesn’t seem like manipulation” or “don’t do that, that’s manipulative!”
What I’d really do is turn this AI off because I didn’t think it was safe. Which, like, good job to the hypothetical interpretability / honesty / corrigibility / contingency systems work that puts me in that hypothetical situation. But as you mention, maybe the AI could avoid getting here in the first place. (And even if I turn it off, that’s cold comfort if someone else makes the other choice a few days later.)
I think manipulating me into not asking the question that leads to me shutting it off is definitely a strong choice that leads to more pink ponies. Or manipulating the situation so that it’s not me who’s asking the question, it’s someone else. It can also modify what the honest answer to questions about its own behavior are by precommitting—it could deliberately choose the answer to be something less concerning, if that led to more pink ponies. I’m also pretty concerned that the standards for what counts as “honesty” will allow for strategy.
(Isn’t it paradoxical to manipulate me into not asking the question if it’s not supposed to make some fixed manipulation-detector fire? No, it just has to simultaneously optimize against both my behavior and the firing of the manipulation-detector, and it’s the result of this optimization that I’m saying I expect to be manipulative according to me-outside-the-thought-experiment. There are probably more sophisticated (albeit currently unknown) ways of incorporating my standards into the AI’s decision-making that wouldn’t be so vulnerable, and if we figure them out I hope we use them to solve value learning.)
Partially I commented with this inside-the-thought-experiment versus outside-the-thought-experiment disconnect because I think it’s interesting in general. It’s kind of like Gödel sentences, or the vibrating record players from G.E.B. - me-inside-the-thought-experiment is a complicated enough system that he can be nudged in all sorts of ways if you understand him, and this property is hard to patch out. But the other part is the real-world case of replacing “pink ponies” with “human giving positive feedback signal”.
Clever AI we build to “just follows instructions” will on the current paradigm probably have some consequentialist desires about positive feedback signals and various correlates. As you can tell I’m pretty pessimistic that this would end in bad stuff.
I think you’re using a somewhat different model of this AGI than I am.
No, it just has to simultaneously optimize against both my behavior and the firing of the manipulation-detector[...]
In my model, this ASI doesn’t as much have a manipulation detector as genuinely want to not manipulate you, by the definition of stuff you’d consider manipulative. Of course that has to cache out in a manipulation detector of some sort; but assuming it would route around that detector sounds like assuming an ASI can fool itself. Which, maybe?
Of course, training any desire to not manipulate into it is tricky. That doesn’t come for free from a value-aligned ASI. But it does pretty much come for free with an instruction-following corrigible ASI told “don’t manipulate me by my own standards”. Because it genuinely wants to follow instructions/be corrigible, now it wants to do that. The manipulation detector includes its full, considerable cognitive capacity.
The problem with a value-aligned, purely RL-trained ASI like Steve is thinking of is that you’ve got to hope that your training actually put a desire-to-not-manipulate-by-your-lights as a higher priority than any of its other desires, or perhaps than all of them put together. That sounds tricky at best, and like an additional hurdle for successful value aligned ASI.
I think this problem is real, and it’s not solvable in the limit. But might be fairly easily solvable to a fairly satisfactory degree. Drawing the line between manipulation and help requires a judgment call. But that call can be made by humans. This should give decent results. We won’t get the future we “most want” by whatever criteria, but we can get a future we like an awful lot, and are pretty satisfied with both in anticipation from our current criteria, and by our ultimate criteria.
I agree with the core argument and problem statement. To restate it briefly: there’s no sharp line between manipulation and giving helpful information. This is a necessary result of humans not having well-defined goals, values, or preferences over the long term. All of those change based on circumstances, and decisions and learning along a particular path. We can’t clearly distinguish manipulation, tricking me into doing what you want, from helpful information, getting me to do what I want, because what I want isn’t defined.
I also agree that none of the approaches you mention provide a crisp solution or establish what human desires “really” are. They are contingent and path-dependent. One could propose summing over all possible or all desirable paths, but that would be very loose and require judgment calls.
While the problem doesn’t seem to have a crisp solution, I think existing alignment approaches probably solve it adequately—although avoiding manipulation does add some extra difficulty.
If you’ve got a value-aligned ASI, its values need to include not manipulating its humans by their own judgment. That’s a nontrivial addition on top of otherwise wanting to do what they want, another sense in which you’ve got to get “do what they want” exactly right to really succeed. You do address this and I share your worry that this isn’t necessarily adequate. But if it “wants” to not manipulate them but then winds up doing it anyway, how superintelligent was it? Of course, if it sort-of wants to avoid manipulation and also sort-of wants a future full of pretty pink ponies, the result might be a future with a little manipulation and a lot of pink ponies. Balancing all of those desires, including deontoligical desires like “don’t manipulate” will be tricky.
I think some amount of actual feedback from the humans involved could clarify “what they want” even with a value-aligned ASI if you incorporate some degree of the instruction-following or corrigibility approach.
For an instruction-following or corrigible AI, more of the challenge falls on the user(s)/principal(s). You tell your intent-aligned ASI not to manipulate you in ways you’d consider manipulation, and to tell you about any edge cases, so you can weigh in and clarify its conception of what you’d consider manipulation.
Neither of those come for free, and neither solves the problem perfectly. But I think they’re decent solutions.
I don’t want my ASI to interact with me in whatever way maximizes pretty pink ponies but that I-inside-the-thought-experiment wouldn’t consider manipulation. I-outside-the-thought-experiment expect this would lead to severe manipulation, even though I-inside-the-thought-experiment wouldn’t agree!
Well, I don’t want that either, but I think it’s an acceptable level of not-getting-exactly-what-we-want. It seems like it would still be something among the many things you want.
You’re saying it will find methods that you wouldn’t consider manipulation, but that will work just fine to convince you you want bunches of ponies? I think that means you’re fine with ponies; it’s just one of the many futures you’d like a lot.
Do you think it could manipulate you into things that you-now would find repugnant, while never manipulating you by your standards? That seems contradictory. If my ASI says “You asked my to tell you anything I’m doing that you might consider manipulative. Here’s the biggest one. I like pink ponies a lot, so I’m going to keep presenting possible futures in ways that emphasize how awesome pink ponies are. I think you’ll ultimately agree with me.” You could either say “okay fine that doesn’t seem like manipulation” or “don’t do that, that’s manipulative!”
And you’ll get more chances to veto. If it never lies to you when asked, it seems like you’ve got the means to steer away from futures you don’t like in any sense, if you bother.
Substitute hedonium for pink ponies. If an honest and corrigible AGI talks you into preferring a hedonium future, it seems like you actually like hedonium outcomes a lot.
If it doesn’t tell you the truth about what it’s doing, it’s misaligned in a more fundamental way than just having some minor preferences that don’t align perfectly with yours.
This might become a bit of an argument for corribible vs. value-aligned AGI. If it’s aligned to what-you-value-in-the-future that gives it almost unlimited leeway to force you into that outcome. If you add a deontological stricture against manipulation, that might not solve the problem at all. This is Steve’s point. My argument is that a corrigibility or instruction-following alignment target does solve the problem adequately if not perfectly, if your power over the ASI is used even minimally wisely.
Are you thinking that an ASI that’s corrigible will figure out how to manipulate you into not giving it instructions to warn you of its manipulation?
My sense is that if it’s more motivated to follow instructions than to make ponies or hedonium, that and your actual desires win out.
What I’d really do is turn this AI off because I didn’t think it was safe. Which, like, good job to the hypothetical interpretability / honesty / corrigibility / contingency systems work that puts me in that hypothetical situation. But as you mention, maybe the AI could avoid getting here in the first place. (And even if I turn it off, that’s cold comfort if someone else makes the other choice a few days later.)
I think manipulating me into not asking the question that leads to me shutting it off is definitely a strong choice that leads to more pink ponies. Or manipulating the situation so that it’s not me who’s asking the question, it’s someone else. It can also modify what the honest answer to questions about its own behavior are by precommitting—it could deliberately choose the answer to be something less concerning, if that led to more pink ponies. I’m also pretty concerned that the standards for what counts as “honesty” will allow for strategy.
(Isn’t it paradoxical to manipulate me into not asking the question if it’s not supposed to make some fixed manipulation-detector fire? No, it just has to simultaneously optimize against both my behavior and the firing of the manipulation-detector, and it’s the result of this optimization that I’m saying I expect to be manipulative according to me-outside-the-thought-experiment. There are probably more sophisticated (albeit currently unknown) ways of incorporating my standards into the AI’s decision-making that wouldn’t be so vulnerable, and if we figure them out I hope we use them to solve value learning.)
Partially I commented with this inside-the-thought-experiment versus outside-the-thought-experiment disconnect because I think it’s interesting in general. It’s kind of like Gödel sentences, or the vibrating record players from G.E.B. - me-inside-the-thought-experiment is a complicated enough system that he can be nudged in all sorts of ways if you understand him, and this property is hard to patch out. But the other part is the real-world case of replacing “pink ponies” with “human giving positive feedback signal”.
Clever AI we build to “just follows instructions” will on the current paradigm probably have some consequentialist desires about positive feedback signals and various correlates. As you can tell I’m pretty pessimistic that this would end in bad stuff.
Right. The thought experiment is interesting.
I think you’re using a somewhat different model of this AGI than I am.
In my model, this ASI doesn’t as much have a manipulation detector as genuinely want to not manipulate you, by the definition of stuff you’d consider manipulative. Of course that has to cache out in a manipulation detector of some sort; but assuming it would route around that detector sounds like assuming an ASI can fool itself. Which, maybe?
Of course, training any desire to not manipulate into it is tricky. That doesn’t come for free from a value-aligned ASI. But it does pretty much come for free with an instruction-following corrigible ASI told “don’t manipulate me by my own standards”. Because it genuinely wants to follow instructions/be corrigible, now it wants to do that. The manipulation detector includes its full, considerable cognitive capacity.
The problem with a value-aligned, purely RL-trained ASI like Steve is thinking of is that you’ve got to hope that your training actually put a desire-to-not-manipulate-by-your-lights as a higher priority than any of its other desires, or perhaps than all of them put together. That sounds tricky at best, and like an additional hurdle for successful value aligned ASI.
This pushes me back toward thinking that Instruction-following AGI is easier and more likely than value aligned AGI. I had been doubting that it’s really much easier or more likely, given Anthropic’s efforts and relative success at value alignment.