What’s a useful notion of the “degree” of misspecification in reward functions?
One worry around labs scaling up RL is that rewards end up being misspecified and this generalises to other kinds of misalignment. But clearly ‘misspecification’ is not a binary variable, it has degrees. If you could formally state that a reward function is misspecified to a ‘smaller degree’, this might let you bound the amount of misalignment you can expect out of doing RL on this.
However, actually formalizing this seems pretty conceptually fiddly. What follows is my initial attempt at doing this, though I didn’t reach a conclusion.
---
Consider the space of reward functions that assign either 0 or 1 to a rollout. If R1 and R2 never assign 1 to the same rollout (so their support is completely disjoint) then you probably expect them to have quite different effects. Whereas if R3 and R4 differ only on 1% of rollouts then maybe they are quite similar.
So one naive notion is “% of rollouts on which they get the same score.”
But this is probably too simplistic, e.g. if R3 and R4 only differ on 1% of rollouts but those rollouts exhibit a consistent pattern then there might still be a noticeable difference in the kind of behaviour they encourage (concretely, you get conditional misalignment or other backdoors)
Also this uniform averaging over samples doesn’t account for the fact that some rollouts are more likely than others, some rollouts might be especially critical for learning and thus updated more on, etc.
It’d be ideal if there was some principled definition that follows cleanly from first principles. Further research may be needed.
Have you looked at a Cohen’s kappa based system. Treat R1 and R2 as raters scoring rollouts. Weighted kappa lets you assign higher cost to disagreements on rollouts that matter more (high-stakes, high-update, etc.), which addresses your point about some rollouts being more critical for learning. And Fleiss’s kappa generalizes to N reward functions if you want to measure agreement across an ensemble rather than just a pair.
The open question is whether κ(R_true, R_proxy) bounds policy regret in the way that EPIC/STARC metrics do for continuous rewards. My intuition is yes for discrete/categorical reward signals (preference judgments, pass/fail), where the L2-family metrics those papers use aren’t natural anyway. Happy to discuss, working on empirical substrate for this.
> So one naive notion is “% of rollouts on which they get the same score.”
rollouts from which policy? In RL lingo, you can talk about something like # of (s,a) pairs on which the rewards differ which marginalizes out the policy (though I am not sure how instructive such formulation would be for LLMs given dynamics are kind of implicitly specified by the LLM itself). You could then talk about differences in occupancy measures induced by the different reward functions as a “policy space” dual of “differences in reward functions”.
What’s a useful notion of the “degree” of misspecification in reward functions?
One worry around labs scaling up RL is that rewards end up being misspecified and this generalises to other kinds of misalignment. But clearly ‘misspecification’ is not a binary variable, it has degrees. If you could formally state that a reward function is misspecified to a ‘smaller degree’, this might let you bound the amount of misalignment you can expect out of doing RL on this.
However, actually formalizing this seems pretty conceptually fiddly. What follows is my initial attempt at doing this, though I didn’t reach a conclusion.
---
Consider the space of reward functions that assign either 0 or 1 to a rollout. If R1 and R2 never assign 1 to the same rollout (so their support is completely disjoint) then you probably expect them to have quite different effects. Whereas if R3 and R4 differ only on 1% of rollouts then maybe they are quite similar.
So one naive notion is “% of rollouts on which they get the same score.”
But this is probably too simplistic, e.g. if R3 and R4 only differ on 1% of rollouts but those rollouts exhibit a consistent pattern then there might still be a noticeable difference in the kind of behaviour they encourage (concretely, you get conditional misalignment or other backdoors)
Also this uniform averaging over samples doesn’t account for the fact that some rollouts are more likely than others, some rollouts might be especially critical for learning and thus updated more on, etc.
It’d be ideal if there was some principled definition that follows cleanly from first principles. Further research may be needed.
Have you looked at a Cohen’s kappa based system. Treat R1 and R2 as raters scoring rollouts. Weighted kappa lets you assign higher cost to disagreements on rollouts that matter more (high-stakes, high-update, etc.), which addresses your point about some rollouts being more critical for learning. And Fleiss’s kappa generalizes to N reward functions if you want to measure agreement across an ensemble rather than just a pair.
The open question is whether κ(R_true, R_proxy) bounds policy regret in the way that EPIC/STARC metrics do for continuous rewards. My intuition is yes for discrete/categorical reward signals (preference judgments, pass/fail), where the L2-family metrics those papers use aren’t natural anyway. Happy to discuss, working on empirical substrate for this.
https://arxiv.org/abs/2209.13085 is probably a good point to start.
> So one naive notion is “% of rollouts on which they get the same score.”
rollouts from which policy? In RL lingo, you can talk about something like # of (s,a) pairs on which the rewards differ which marginalizes out the policy (though I am not sure how instructive such formulation would be for LLMs given dynamics are kind of implicitly specified by the LLM itself). You could then talk about differences in occupancy measures induced by the different reward functions as a “policy space” dual of “differences in reward functions”.