On one hand, if you positively reinforce, the system will seek it out, if you negatively reinforce the system will work around it.
On the other hand, there doesn’t seem to be a principled difference between positive reinforcement and negative reinforcement. Like I would assume that the zero point wouldn’t affect the trade-off between two actions as long as the difference was fixed.
Having thought about it a bit more, I think I managed to resolve the tension. It seems that if at least one of the actions is positive utility, then the system has a reason to maneuver you into a hypothetical state where you choose between them, while if both are negative utility then the system has a reason to actively steer you away from having to make such a choice.
(This analysis is still naive in that it doesn’t account for opportunity cost).
I’d really love to see greater formalisation of this intuition. Even what I’ve said above is quite ambiguous.
On the other hand, there doesn’t seem to be a principled difference between positive reinforcement and negative reinforcement. Like I would assume that the zero point wouldn’t affect the trade-off between two actions as long as the difference was fixed.
This is only true for optimal policies, no? For learned policies, positive reward will upweight and generalize certain circuits (like “approach juice”), while negative reward will downweight and generally-discourage those same circuits. This can then lead to path-dependent differences in generalization (e.g. whether person pursues juice in general).
(That said, it seems like to useful check to see what the optimal policy will do. And if someone believes it won’t achieve the optimal policy, it seems useful to try to understand the barrier that stops that. I don’t feel quite clear on this yet).
My initial thoughts were:
On one hand, if you positively reinforce, the system will seek it out, if you negatively reinforce the system will work around it.
On the other hand, there doesn’t seem to be a principled difference between positive reinforcement and negative reinforcement. Like I would assume that the zero point wouldn’t affect the trade-off between two actions as long as the difference was fixed.
Having thought about it a bit more, I think I managed to resolve the tension. It seems that if at least one of the actions is positive utility, then the system has a reason to maneuver you into a hypothetical state where you choose between them, while if both are negative utility then the system has a reason to actively steer you away from having to make such a choice.
(This analysis is still naive in that it doesn’t account for opportunity cost).
I’d really love to see greater formalisation of this intuition. Even what I’ve said above is quite ambiguous.
This is only true for optimal policies, no? For learned policies, positive reward will upweight and generalize certain circuits (like “approach juice”), while negative reward will downweight and generally-discourage those same circuits. This can then lead to path-dependent differences in generalization (e.g. whether person pursues juice in general).
(In general, I think reward is not best understood as an optimization target like “utility.”)
Good point.
(That said, it seems like to useful check to see what the optimal policy will do. And if someone believes it won’t achieve the optimal policy, it seems useful to try to understand the barrier that stops that. I don’t feel quite clear on this yet).