I agree RLVR isn’t exactly the thing I want to gesture at, the thing I more want to gesture at is something like “RL being used a bunch in post-training, that’s neither RLHF nor Constitutional AI, as centrally defined.” To be clear I think RLHF or RLAIF via Constitutional AI has their own pathologies as well[1], but I think the degree of reward-hacking and deception are mostly not downstream of that.
I agree RLVR isn’t exactly the thing I want to gesture at, the thing I more want to gesture at is something like “RL being used a bunch in post-training, that’s neither RLHF nor Constitutional AI, as centrally defined.” To be clear I think RLHF or RLAIF via Constitutional AI has their own pathologies as well[1], but I think the degree of reward-hacking and deception are mostly not downstream of that.
eg sycophancy, psychosis