Thanks for the post Jan, many of your points really resonated with me!
Do you think something like RLVH (Reinforcement Learning with Verifiable Hacks) could help move the needle on some the issues you outlined? While it doesn’t catch all classes of hacks, adding impossible tests to RL environments seems like a reliable way to flag some types of hacking. If these tests start passing, flag and disincentive the rollouts. And similarly, for LLM Judges, add rubric categories that should be impossible to satisfy without cheating or manipulating the judge. We could randomize the frequency, density, and types of honeypots injected. [Extending the work of ImpossibleBench] Also we could manipulate the questions to be impossible instead of manipulating the answers (although not sure how much that would gain us, it might just make the LLM good at detecting impossible questions). We’d have to carefully monitor for generalization either way. I think the real risk is we push the classes of hacks to be less detectable or even more conditional. But maybe good randomization, Inoculation, and other regularizers could help get more generalization out of the disincentives?
Thanks for the post Jan, many of your points really resonated with me!
Do you think something like RLVH (Reinforcement Learning with Verifiable Hacks) could help move the needle on some the issues you outlined? While it doesn’t catch all classes of hacks, adding impossible tests to RL environments seems like a reliable way to flag some types of hacking. If these tests start passing, flag and disincentive the rollouts. And similarly, for LLM Judges, add rubric categories that should be impossible to satisfy without cheating or manipulating the judge. We could randomize the frequency, density, and types of honeypots injected. [Extending the work of ImpossibleBench] Also we could manipulate the questions to be impossible instead of manipulating the answers (although not sure how much that would gain us, it might just make the LLM good at detecting impossible questions). We’d have to carefully monitor for generalization either way. I think the real risk is we push the classes of hacks to be less detectable or even more conditional. But maybe good randomization, Inoculation, and other regularizers could help get more generalization out of the disincentives?
I don’t know what’s RLVH unfortunately.