In regular finetuning, always training it to produce often strongly biases it toward . Perhaps this similarly happens with ACT/BCT? (On all remaining inputs, ACT/BCT finetune it to produce , or to match its activations in the latter). Then, possibly much of the effect of RMCT is not the specific rate but the result of mixing in some non-?
I think this is a genuine concern with BCT/ACT when we have more than 2 related inputs for the same samples (eg., multiple biases, such as in Appendix F of the paper), but not the main paper results, which only train on 1 biased setting. The result I’d expect here is mode collapse, where the LLM is trained to output a narrower distribution of responses, which RMCT largely avoids (it arguably still trains on a narrower distribution than the control since rate matching may select for more similar responses).
In regular finetuning, always training it to produce
Thanks for the comment!
I think this is a genuine concern with BCT/ACT when we have more than 2 related inputs for the same samples (eg., multiple biases, such as in Appendix F of the paper), but not the main paper results, which only train on 1 biased setting. The result I’d expect here is mode collapse, where the LLM is trained to output a narrower distribution of responses, which RMCT largely avoids (it arguably still trains on a narrower distribution than the control since rate matching may select for more similar responses).