I think AIS could benefit from “forward-deployed research engineers”.
Consider that:
Within AIS, we often talk about “differentially accelerating safety” (as opposed to capabilities)
Automation can provide extreme uplift (as reported by Boris Cherny, anyway). Also, the ceiling on automation climbs extremely rapidly with time (e.g. METR time horizons) and has not plateaued yet.
However, the benefits have yet to diffuse widely into society at large.
Some of this is irreducible (e.g. some tasks are hard to automate)
But some is reducible (e.g. orgs just don’t know how / haven’t allocated time etc)
I’d guess there are a bunch of organizations which are doing good work now + have yet to fully benefit from automation, and so could do a lot more good work
So: plausibly a pretty good thing to do is to just hire a large number of research engineers, who have research engineering + automation skills, and “forward deploy” them at orgs which would benefit from being sped up. Alternatively, have consulting services which help orgs implement best practices.
Corollary: It might be a good idea to create new orgs which offer such “forward deployed research engineer” services (AE studio comes to mind, but IMO we could use more)
where would you see these AI safety forward deployed engineers mostly work at?
from your description, I believe they have the highest value in model labs as we are talking about AIS wrt to catastrophic risks, but there aren’t really that many labs — and I don’t think Chinese labs would hire these engineers from US/UK/EU.
I see that a lot of startups are hiring for security research engineers (eg Mirendil that focuses on AI R&D that accelerates science) which I believe that’s having a higher demand at the moment but my understanding is that theres a higher focus on alignment researchers rather than security researchers within LW community (at least based on my read on programs like MATS)
I think forward-deployed research engineers (FDREs) should mainly work with independent nonprofits / academic labs who have a good theory of change but wouldn’t have capital to normally attract RE talent by themselves
I think there should be conditions attached to the use of such research engineers, e.g. working towards an agreed agenda, and making sure to publish all findings in an unbiased way.
In practice FDREs should be viewed as another type of “grant”, which is somewhat-fungible (but not totally fungible) with monetary grants, and doled out by equivalent grantmaking organizations
I believe they have the highest value in model labs
I disagree with this, for the following reasons
Even assuming that catastrophic risks are the main category of risk, the best way to address this might not be by accelerating labs. I subscribe to Geoffrey Irving’s view that more diverse bets are needed to solve alignment.
It’s not clear that having these engineers be at labs would differentially accelerate safety, and indeed it might accelerate capabilities instead (e.g. research is dual use, or labs can respond by allocating more of their own engineer-hours to capabilities instead of safety / alignment research)
I think labs are already able to attract sufficient talent so they don’t need additional help from the nonprofit side
my understanding is that theres a higher focus on alignment researchers rather than security researchers (at least based on my read on programs like MATS)
I also disagree with this, MATS seems to be diversifying quite hard with their recent batches, and I would now say that (prosaic alignment, conceptual alignment, cybersecurity, policy / governance, biosecurity, and founding / field-building) are all represented to some extent
I think AIS could benefit from “forward-deployed research engineers”.
Consider that:
Within AIS, we often talk about “differentially accelerating safety” (as opposed to capabilities)
Automation can provide extreme uplift (as reported by Boris Cherny, anyway). Also, the ceiling on automation climbs extremely rapidly with time (e.g. METR time horizons) and has not plateaued yet.
However, the benefits have yet to diffuse widely into society at large.
Some of this is irreducible (e.g. some tasks are hard to automate)
But some is reducible (e.g. orgs just don’t know how / haven’t allocated time etc)
I’d guess there are a bunch of organizations which are doing good work now + have yet to fully benefit from automation, and so could do a lot more good work
So: plausibly a pretty good thing to do is to just hire a large number of research engineers, who have research engineering + automation skills, and “forward deploy” them at orgs which would benefit from being sped up. Alternatively, have consulting services which help orgs implement best practices.
Corollary: It might be a good idea to create new orgs which offer such “forward deployed research engineer” services (AE studio comes to mind, but IMO we could use more)
where would you see these AI safety forward deployed engineers mostly work at?
from your description, I believe they have the highest value in model labs as we are talking about AIS wrt to catastrophic risks, but there aren’t really that many labs — and I don’t think Chinese labs would hire these engineers from US/UK/EU.
I see that a lot of startups are hiring for security research engineers (eg Mirendil that focuses on AI R&D that accelerates science) which I believe that’s having a higher demand at the moment but my understanding is that theres a higher focus on alignment researchers rather than security researchers within LW community (at least based on my read on programs like MATS)
I think forward-deployed research engineers (FDREs) should mainly work with independent nonprofits / academic labs who have a good theory of change but wouldn’t have capital to normally attract RE talent by themselves
I think there should be conditions attached to the use of such research engineers, e.g. working towards an agreed agenda, and making sure to publish all findings in an unbiased way.
In practice FDREs should be viewed as another type of “grant”, which is somewhat-fungible (but not totally fungible) with monetary grants, and doled out by equivalent grantmaking organizations
I disagree with this, for the following reasons
Even assuming that catastrophic risks are the main category of risk, the best way to address this might not be by accelerating labs. I subscribe to Geoffrey Irving’s view that more diverse bets are needed to solve alignment.
It’s not clear that having these engineers be at labs would differentially accelerate safety, and indeed it might accelerate capabilities instead (e.g. research is dual use, or labs can respond by allocating more of their own engineer-hours to capabilities instead of safety / alignment research)
I think labs are already able to attract sufficient talent so they don’t need additional help from the nonprofit side
I also disagree with this, MATS seems to be diversifying quite hard with their recent batches, and I would now say that (prosaic alignment, conceptual alignment, cybersecurity, policy / governance, biosecurity, and founding / field-building) are all represented to some extent