When told to raise the score on a task, the models will sometimes brazenly cheat. It seems difficult to predict when this will happen vs. when the run will go smoothly.
Does this persist when using hardcoded metrics where “cheating” isn’t really meaningful? For instance, using an LLM to interactively make e.g. a Lean specification of some alignment-relevant result, and then sending off auto-research in a separate session to generate some code-snippet or similar whose structure and execution-in-a-separate-sandbox can be formalized and checked against the proof criteria?
It seems the above would reduce both uninterpretability and metric-gaming, since the metric exactly-equals the spirit of the task and is in-theory not amenable to judge-hacking. (Resolution is investigating formal-reasoning for automated alignment, I believe. I’ve also started looking at making Claude Code build a toy scaffold to test this approach.)
But we’ve found that just asking the model how it feels about the work it’s doing is surprisingly effective.
If models are less likely to lie when acting as their own LLM-judges, then could you just give the research LLM a natural language intent including instructions to consistently score itself and calibrate its scoring-criteria for metrics and prioritization purposes? Possibly the lack of an external judge would reduce reward-hacking, from a prompt-engineering perspective?
Hillclimbing metrics are often off-target from the spirit of an alignment task.
You might find this recent post of interest, w.r.t. LLMs showing sometimes-drastically different behaviors w.r.t. alignment to user intent in explicitly-graded contexts, at least if they’re made aware of the scoring.
Is it tractable to have binary / qualitative criteria and score-responses for the actual runs, and use human-designed or LLM-as-judge numeric scoring purely for metrics/dashboards, where any LLM with authority to change the judging criteria is human-controlled to ensure intent-alignment? I’ve had some success with using this pattern to increase response-quality, though not specifically in a research-harness context.
Does this persist when using hardcoded metrics where “cheating” isn’t really meaningful? For instance, using an LLM to interactively make e.g. a Lean specification of some alignment-relevant result, and then sending off auto-research in a separate session to generate some code-snippet or similar whose structure and execution-in-a-separate-sandbox can be formalized and checked against the proof criteria?
It seems the above would reduce both uninterpretability and metric-gaming, since the metric exactly-equals the spirit of the task and is in-theory not amenable to judge-hacking. (Resolution is investigating formal-reasoning for automated alignment, I believe. I’ve also started looking at making Claude Code build a toy scaffold to test this approach.)
If models are less likely to lie when acting as their own LLM-judges, then could you just give the research LLM a natural language intent including instructions to consistently score itself and calibrate its scoring-criteria for metrics and prioritization purposes? Possibly the lack of an external judge would reduce reward-hacking, from a prompt-engineering perspective?
You might find this recent post of interest, w.r.t. LLMs showing sometimes-drastically different behaviors w.r.t. alignment to user intent in explicitly-graded contexts, at least if they’re made aware of the scoring.
Is it tractable to have binary / qualitative criteria and score-responses for the actual runs, and use human-designed or LLM-as-judge numeric scoring purely for metrics/dashboards, where any LLM with authority to change the judging criteria is human-controlled to ensure intent-alignment? I’ve had some success with using this pattern to increase response-quality, though not specifically in a research-harness context.