What do you think about distribution matching algorithms like GFlowNets as a safer option? E.g. this paper in 2023 proposed training LLMs using RL with a GFlowNets objective function?
As far as I understand, this method argues that SFT and RL are maximizing training signals (SFT is maximizing likelihood, RL is maximizing rewards), which lead to mode collapse because models are optimized towards only these handful of maximized modes, while a distribution matching paradigm is better at learning a model that better preserves the structure of the information being learned.
So perhaps distribution matching will avoid “ruthlessly maximize reward functions, including in ways the programmer obviously didn’t intend”, or avoid optimizing against those “pathological craziness”? (although I also see this method leading to the bad direction since optimizing for a distribution seems to expose many more modes that are adjacent to those “pathological craziness”?)
What do you think about distribution matching algorithms like GFlowNets as a safer option? E.g. this paper in 2023 proposed training LLMs using RL with a GFlowNets objective function?
As far as I understand, this method argues that SFT and RL are maximizing training signals (SFT is maximizing likelihood, RL is maximizing rewards), which lead to mode collapse because models are optimized towards only these handful of maximized modes, while a distribution matching paradigm is better at learning a model that better preserves the structure of the information being learned.
So perhaps distribution matching will avoid “ruthlessly maximize reward functions, including in ways the programmer obviously didn’t intend”, or avoid optimizing against those “pathological craziness”? (although I also see this method leading to the bad direction since optimizing for a distribution seems to expose many more modes that are adjacent to those “pathological craziness”?)
I don’t know much about GFlowNets. Anyway, that paper seems very specific to LLMs, which are off-topic for this post (see Q1 & Q7), sorry.