alignment of strong optimizers simply cannot be done without grounding out in something fundamentally different from a feedback signal.
I don’t think this is obvious at all. Essentially, we have to make sure that humans give feedback that matches their preferences, and that the agent isn’t changing the human’s preferences to be more easily optimized.
We have the following tools at our disposal:
Recursive reward modelling / Debate. By training agents to help with feedback, improvements in optimization power boosts both the feedback and the process potentially fooling the feedback. It’s possible that it’s easier to fool humans than it is to help them not be fooled, but it’s not obvious this is the case.
Path-specific objectives. By training an explicit model of how humans will be influenced by agent behavior, we can design an agent that optimizes the hypothetical feedback that would have been given, had the agent’s behavior not changed the human’s preferences (under some assumptions).
This makes me mildly optimistic of using feedback even for relatively powerful optimization.
Essentially, we have to make sure that humans give feedback that matches their preferences...
Humans’ stated preferences do not match their preferences-in-hindsight, neither of those matches humans’ self-reported happiness/satisfaction in-the-moment, none of that matches humans’ revealed preferences, and all of those are time-inconsistent. IIRC the first section of Kahnemann’s textbook Well Being: The Foundations of Hedonic Psychology is devoted entirely to the problem of getting feedback from humans on what they actually like, and the tldr is “people have been working on this for decades and all our current proxies have known problems” (not to say they don’t have unknown problems too, but they definitely have known problems). Once we get past the basic proxies, we pretty quickly run into fundamental conceptual issues about what we even mean by “human preferences”.
Make sure the agent isn’t changing the preferences
It seems that RRM/Debate somewhat addresses both of these, and path-specific objectives is mainly aimed at addressing issue 2. I think (part of) John’s point is that RRM/Debate don’t address issue 1 very well, because we don’t have very good or robust processes for judging the various ways we could construct or improve these schemes. Debate relies on a trustworthy/reliable judge at the end of the day, and we might not actually have that.
If the problem is “humans don’t give good feedback”, then we can’t directly train agents to “help” with feedback; there’s nothing besides human feedback to give a signal of what’s “helping” in the first place. We can choose some proxy for what we think is helpful, but then that’s another crappy proxy which will break down under optimization pressure.
It’s not just about “fooling” humans, though that alone is a sufficient failure mode. Bear in mind that in order for “helping humans not be fooled” to be viable as a primary alignment strategy it must be the case that it’s easier to help humans not be fooled than to fool them in approximately all cases, because otherwise a hostile optimizer will head straight for the cases where humans are fallible. And I claim it is very obvious, from looking at existing real-world races between those trying to deceive and those trying to expose the deception, that there will be plenty of cases where the expose-deception side does not have a winning strategy.
The agent changing “human preferences” is another sufficient failure mode. The strategy of “design an agent that optimizes the hypothetical feedback that would have been given” is indeed a conceptually-valid way to solve that problem, and is notably not a direct feedback signal in the RL sense. At that point, we’re doing EU maximization, not reinforcement learning. We’re optimizing for expected utility from a fixed model, we’re not optimizing a feedback signal from the environment. Of course a bunch of the other problems of human feedback still carry over; “the hypothetical feedback a human would have given” is still a crappy proxy. But it’s a step in the right direction.
Sure, humans are sometimes inconsistent, and we don’t always know what we want (thanks for the references, that’s useful!). But I suspect we’re mainly inconsistent in borderline cases, which aren’t catastrophic to get wrong. I’m pretty sure humans would reliably state that they don’t want to be killed, or that lots of other people die, etc. And that when they have a specific task in mind , they state that they want the task done rather than not. All this subject to that they actually understand the main considerations for whatever plan or outcome is in question, but that is exactly what debate and rrm are for
I don’t think this is obvious at all. Essentially, we have to make sure that humans give feedback that matches their preferences, and that the agent isn’t changing the human’s preferences to be more easily optimized.
We have the following tools at our disposal:
Recursive reward modelling / Debate. By training agents to help with feedback, improvements in optimization power boosts both the feedback and the process potentially fooling the feedback. It’s possible that it’s easier to fool humans than it is to help them not be fooled, but it’s not obvious this is the case.
Path-specific objectives. By training an explicit model of how humans will be influenced by agent behavior, we can design an agent that optimizes the hypothetical feedback that would have been given, had the agent’s behavior not changed the human’s preferences (under some assumptions).
This makes me mildly optimistic of using feedback even for relatively powerful optimization.
Minor rant about this is particular:
Humans’ stated preferences do not match their preferences-in-hindsight, neither of those matches humans’ self-reported happiness/satisfaction in-the-moment, none of that matches humans’ revealed preferences, and all of those are time-inconsistent. IIRC the first section of Kahnemann’s textbook Well Being: The Foundations of Hedonic Psychology is devoted entirely to the problem of getting feedback from humans on what they actually like, and the tldr is “people have been working on this for decades and all our current proxies have known problems” (not to say they don’t have unknown problems too, but they definitely have known problems). Once we get past the basic proxies, we pretty quickly run into fundamental conceptual issues about what we even mean by “human preferences”.
The desiderata you mentioned:
Make sure the feedback matches the preferences
Make sure the agent isn’t changing the preferences
It seems that RRM/Debate somewhat addresses both of these, and path-specific objectives is mainly aimed at addressing issue 2. I think (part of) John’s point is that RRM/Debate don’t address issue 1 very well, because we don’t have very good or robust processes for judging the various ways we could construct or improve these schemes. Debate relies on a trustworthy/reliable judge at the end of the day, and we might not actually have that.
If the problem is “humans don’t give good feedback”, then we can’t directly train agents to “help” with feedback; there’s nothing besides human feedback to give a signal of what’s “helping” in the first place. We can choose some proxy for what we think is helpful, but then that’s another crappy proxy which will break down under optimization pressure.
It’s not just about “fooling” humans, though that alone is a sufficient failure mode. Bear in mind that in order for “helping humans not be fooled” to be viable as a primary alignment strategy it must be the case that it’s easier to help humans not be fooled than to fool them in approximately all cases, because otherwise a hostile optimizer will head straight for the cases where humans are fallible. And I claim it is very obvious, from looking at existing real-world races between those trying to deceive and those trying to expose the deception, that there will be plenty of cases where the expose-deception side does not have a winning strategy.
The agent changing “human preferences” is another sufficient failure mode. The strategy of “design an agent that optimizes the hypothetical feedback that would have been given” is indeed a conceptually-valid way to solve that problem, and is notably not a direct feedback signal in the RL sense. At that point, we’re doing EU maximization, not reinforcement learning. We’re optimizing for expected utility from a fixed model, we’re not optimizing a feedback signal from the environment. Of course a bunch of the other problems of human feedback still carry over; “the hypothetical feedback a human would have given” is still a crappy proxy. But it’s a step in the right direction.
Sure, humans are sometimes inconsistent, and we don’t always know what we want (thanks for the references, that’s useful!). But I suspect we’re mainly inconsistent in borderline cases, which aren’t catastrophic to get wrong. I’m pretty sure humans would reliably state that they don’t want to be killed, or that lots of other people die, etc. And that when they have a specific task in mind , they state that they want the task done rather than not. All this subject to that they actually understand the main considerations for whatever plan or outcome is in question, but that is exactly what debate and rrm are for