This seems like a worthwhile problem to be trying to solve. Not sure how much traction the listed approaches offer for the relevant cases though? Maybe I’m not parsing it quite right. How close to “this holds a mildly superhuman system from misusing free parameters enough to really matter” do you think you are?
It’s hard to measure distance over additional research needed!
To be confident in an unexploitable search method you’d definitely want empirical evidence. And I can imagine empirical evidence strong enough to give confidence that our method helps enough to really matter.
Here’s an evaluation that would give me confidence: Take an adversarially initialized model (e.g. by doing imitation learning on code with vulnerabilities) and train on our game. If the rate at which the model produced code with vulnerabilities returned to baseline (with steps say linear in the number of IL samples) then I’d be fairly confident that our method was useful!
This seems like a worthwhile problem to be trying to solve. Not sure how much traction the listed approaches offer for the relevant cases though? Maybe I’m not parsing it quite right. How close to “this holds a mildly superhuman system from misusing free parameters enough to really matter” do you think you are?
It’s hard to measure distance over additional research needed!
To be confident in an unexploitable search method you’d definitely want empirical evidence. And I can imagine empirical evidence strong enough to give confidence that our method helps enough to really matter.
Here’s an evaluation that would give me confidence: Take an adversarially initialized model (e.g. by doing imitation learning on code with vulnerabilities) and train on our game. If the rate at which the model produced code with vulnerabilities returned to baseline (with steps say linear in the number of IL samples) then I’d be fairly confident that our method was useful!