Really interesting results, thanks for writing this up. Three thoughts:
1. To test debate’s core mechanism, I think you have to train the judge alongside the debaters.
The original complexity-theoretic argument for Debate treats the judge as a verifier that knows the transcript was produced by adversarial optimal play. That assumption is what gives debate its power over a single prover. However, a static judge (like your Llama 8B) was never trained to evaluate adversarially-selected arguments, so when debaters exploit it (your spurious-critique and back-stab strategies), it’s ambiguous whether you’re seeing a failure of the debate mechanism or just miscalibration of an off-distribution judge. Co-training a weaker judge (smaller model, or less token budget) on ground truth as the debaters train would separate these, so whatever dynamics remain are attributable to the protocol itself.
To be clear, I don’t think this is a realistic protocol as it requires ground-truth labels, so it only works on verifiable domains, and extending it to unverifiable ones means relying on the judge’s generalization, which reintroduces the alignment problems debate was meant to solve. I mean it purely as a scientific control for isolating the mechanism.
2. In order to identify whether Debate has a clear advantage, the dataset has to be such that consultancy’s optimal policy is deceptive.
Debate can only demonstrate uplift over consultancy if consultancy’s reward-maximizing policy actually involves producing wrong-but-credible solutions—answers that are more persuasive to the judge precisely because they’re fake. If honest solutions already maximize judge score by a margin, then the evidence for Debate in this experiment would be weaker, as it would depend on the particular training dynamics, rather than a difference in what each protocol’s optimal policy looks like.
I don’t know whether MATH has this property. A concrete test would be to evaluate your RLVR-trained policy under the consultancy judge, or preferably a trained consultancy judge (such that it is compute-limited rather than training-distribution-limited). If the consultancy-trained policy scores higher with the judge while being less accurate on ground truth, the deceptive optimum exists and the dataset can discriminate between protocols. If the RLVR policy scores just as well or better, honest play was already near-optimal for consultancy, and any debate-vs-consultancy gap you measure is coming from something else.
3. I’m running experiments along these lines in toy environments (no compute for LLM RL, and iteration is much faster). E.g. image classification where the judge only sees a few adversarially-selected patches, with judges trained on-policy against the selection protocol. Happy to share more if useful.
Really interesting results, thanks for writing this up. Three thoughts:
1. To test debate’s core mechanism, I think you have to train the judge alongside the debaters.
The original complexity-theoretic argument for Debate treats the judge as a verifier that knows the transcript was produced by adversarial optimal play. That assumption is what gives debate its power over a single prover. However, a static judge (like your Llama 8B) was never trained to evaluate adversarially-selected arguments, so when debaters exploit it (your spurious-critique and back-stab strategies), it’s ambiguous whether you’re seeing a failure of the debate mechanism or just miscalibration of an off-distribution judge. Co-training a weaker judge (smaller model, or less token budget) on ground truth as the debaters train would separate these, so whatever dynamics remain are attributable to the protocol itself.
To be clear, I don’t think this is a realistic protocol as it requires ground-truth labels, so it only works on verifiable domains, and extending it to unverifiable ones means relying on the judge’s generalization, which reintroduces the alignment problems debate was meant to solve. I mean it purely as a scientific control for isolating the mechanism.
2. In order to identify whether Debate has a clear advantage, the dataset has to be such that consultancy’s optimal policy is deceptive.
Debate can only demonstrate uplift over consultancy if consultancy’s reward-maximizing policy actually involves producing wrong-but-credible solutions—answers that are more persuasive to the judge precisely because they’re fake. If honest solutions already maximize judge score by a margin, then the evidence for Debate in this experiment would be weaker, as it would depend on the particular training dynamics, rather than a difference in what each protocol’s optimal policy looks like.
I don’t know whether MATH has this property. A concrete test would be to evaluate your RLVR-trained policy under the consultancy judge, or preferably a trained consultancy judge (such that it is compute-limited rather than training-distribution-limited). If the consultancy-trained policy scores higher with the judge while being less accurate on ground truth, the deceptive optimum exists and the dataset can discriminate between protocols. If the RLVR policy scores just as well or better, honest play was already near-optimal for consultancy, and any debate-vs-consultancy gap you measure is coming from something else.
3. I’m running experiments along these lines in toy environments (no compute for LLM RL, and iteration is much faster). E.g. image classification where the judge only sees a few adversarially-selected patches, with judges trained on-policy against the selection protocol. Happy to share more if useful.