Hot take: reasoning models trained with diverse RL across diverse graders will be steerable by fine-tuning them to assume a narrow distribution of graders. This is analogous to how pretrained models can be instruct-tuned to assume a persona.
The analogy: pretraining teaches a model to simulate and select a writer from a distribution of writers. RL (reasoning training) similarly teaches it to simulate and select a grader from a distribution of graders. So you could steer an LLM by specifying both a persona and a grader within the training distribution.
The upshot is that at the end of RL training, you’d want to train on a narrow distribution of graders that promote the goals you actually care about. This might mean using your strongest, most expensive graders and/or your least gameable training environments at that final stage.
I think this is not important. What is important is the clarity, legibility and content value of the text, not how much of the final writing is human or IA.
I don’t understand the disagreement. How are your points pushing back against “models learning simulation of graders and thus allowing us to steer them by biasing the distribution of simulated graders”.
From an EAG discussion, my understanding is that Apollo Research is going to publish research in which models trained on diverse graders learn to think about their grader to optimise their answer and reward hack. While models trained on non-diverse graders reward hack without talking about their graders. This seems to point to models learning to reason and simulate their grader when they are trained on diverse graders, which can be expected since simulating a distribution of graders is much more training efficient, and loss-reducing than learning to reward hack every single grader.
Steering reasoning models via grader fine-tuning
Hot take: reasoning models trained with diverse RL across diverse graders will be steerable by fine-tuning them to assume a narrow distribution of graders. This is analogous to how pretrained models can be instruct-tuned to assume a persona.
The analogy: pretraining teaches a model to simulate and select a writer from a distribution of writers. RL (reasoning training) similarly teaches it to simulate and select a grader from a distribution of graders. So you could steer an LLM by specifying both a persona and a grader within the training distribution.
The upshot is that at the end of RL training, you’d want to train on a narrow distribution of graders that promote the goals you actually care about. This might mean using your strongest, most expensive graders and/or your least gameable training environments at that final stage.
A plausible (simplified) pipeline: pretraining → HHH fine-tuning → RL(VR) → omniscient-grader fine-tuning.
I think this is directionally correct but not quite right, eg because reward is not the optimization target
I would instead reframe as “RL is likely to select between policies that are already salient to the model”.
(also, noting that this seems AI generated)
I think this is not important. What is important is the clarity, legibility and content value of the text, not how much of the final writing is human or IA.
I don’t understand the disagreement. How are your points pushing back against “models learning simulation of graders and thus allowing us to steer them by biasing the distribution of simulated graders”.
Mainly that I don’t think models simulate their graders, they just adopt policies that get rewarded. See: https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target
To be clear I think the effect is very similar so the disagreement is not super load bearing.
From an EAG discussion, my understanding is that Apollo Research is going to publish research in which models trained on diverse graders learn to think about their grader to optimise their answer and reward hack. While models trained on non-diverse graders reward hack without talking about their graders. This seems to point to models learning to reason and simulate their grader when they are trained on diverse graders, which can be expected since simulating a distribution of graders is much more training efficient, and loss-reducing than learning to reward hack every single grader.
Hmm, okay that could update me a fair bit. I’ll read it when it comes out!
FWIW my current thinking around this has been pretty shaped by shard theory, and in particular this bit: https://www.lesswrong.com/posts/8ccTZ9ZxpJrvnxt4F/shard-theory-in-nine-theses-a-distillation-and-critical#4__The_optimization_target_is_poorly_modeled_by_the_reward_function