The Ten People On The Inside likely should prioritize things that capabilities teams aren’t going to do anyway and people considering joining labs with similar motivations should likely think about whether they’re sure they’re going to be able to do work that wouldn’t otherwise be done for capabilities reasons. The thing that’s top of mind here is reward hacking, which is increasingly becoming a capabilities problem.[1] To the extent research is done here, it seems pretty important to avoid the failure mode of “repeatedly iterate on environments and observable behavior”[2] which is the kind of thing I would expect by default for capabilities reasons.
A good example of research on reward hacking that I don’t think would’ve necessarily gotten done “by default for capabilities reasons” is inoculation prompting and similar investigations
This can take the form of specific safety training targeting the observable behavior, post-hoc environment fixes, or having an iteration loop between your pre-deployment automated alignment evals and your alignment training
The Ten People On The Inside likely should prioritize things that capabilities teams aren’t going to do anyway and people considering joining labs with similar motivations should likely think about whether they’re sure they’re going to be able to do work that wouldn’t otherwise be done for capabilities reasons. The thing that’s top of mind here is reward hacking, which is increasingly becoming a capabilities problem.[1] To the extent research is done here, it seems pretty important to avoid the failure mode of “repeatedly iterate on environments and observable behavior”[2] which is the kind of thing I would expect by default for capabilities reasons.
A good example of research on reward hacking that I don’t think would’ve necessarily gotten done “by default for capabilities reasons” is inoculation prompting and similar investigations
This can take the form of specific safety training targeting the observable behavior, post-hoc environment fixes, or having an iteration loop between your pre-deployment automated alignment evals and your alignment training
Legible vs. Illegible AI Safety Problems somewhat resonates with this.