models trained on diverse graders learn to think about their grader to optimise their answer and reward hack
Hmm, okay that could update me a fair bit. I’ll read it when it comes out!
FWIW my current thinking around this has been pretty shaped by shard theory, and in particular this bit: https://www.lesswrong.com/posts/8ccTZ9ZxpJrvnxt4F/shard-theory-in-nine-theses-a-distillation-and-critical#4__The_optimization_target_is_poorly_modeled_by_the_reward_function
Hmm, okay that could update me a fair bit. I’ll read it when it comes out!
FWIW my current thinking around this has been pretty shaped by shard theory, and in particular this bit: https://www.lesswrong.com/posts/8ccTZ9ZxpJrvnxt4F/shard-theory-in-nine-theses-a-distillation-and-critical#4__The_optimization_target_is_poorly_modeled_by_the_reward_function