Thanks for this (very thorough) answer. I’m especially excited to see that you’ve reached out to 25 AI gov researchers & already have four governance mentors for summer 2024. (Minor: I think the post mentioned that you plan to have at least 2, but it seems like there are already 4 confirmed and you’re open to more; apologies if I misread something though.)
A few quick responses to other stuff:
I appreciate a lot of the other content presented. It feels to me like a lot of it is addressing the claim “it is net positive for MATS to upskill people who end up working at scaling labs”, whereas I think the claims I made were a bit different. (Specifically, I think I was going for more “Do you think this is the best thing for MATS to be focusing on, relative to governance/policy”and “Do you think there are some cultural things that ought to be examined to figure out why scaling labs are so much more attractive than options that at-least-to-me seem more impactful in expectation”).
RE AI control, I don’t think I’m necessarily underestimating its popularity as a metastrategy. I’m broadly aware that a large fraction of the Bay Area technical folks are excited about control. However, I think when characterizing the AI safety community as a whole (not just technical people), the shift toward governance/policy macrostrategies is (much) stronger than the shift toward the control macrostrategy. (Separately, I think I’m more excited about foundational work in AI control that looks more like the kind of thing that Buck/Ryan have written about is separate from typical prosaic work (e.g., interpretability), even though lots of typical prosaic work could be argued to be connected to the control macrostrategy.)
+1 that AI governance mentors might be harder to find for some of the reasons you listed.
Do you think there are some cultural things that ought to be examined to figure out why scaling labs are so much more attractive than options that at-least-to-me seem more impactful in expectation?
As a naive guess, I would consider the main reasons to be:
People seeking jobs in AI safety often want to take on “heroic responsibility.” Work on evals and policy, while essential, might be seen as “passing the buck” onto others, often at scaling labs, who have to “solve the wicked problem of AI alignment/control” (quotes indicate my caricature of a hypothetical person). Anecdotally, I’ve often heard people in-community disparage AI safety strategies that primarily “buy time” without “substantially increasing the odds AGI is aligned.” Programs like MATS emphasizing the importance of AI governance and including AI strategy workshops might help shift this mindset, if it exists.
Roles in AI gov/policy, while impactful at reducing AI risk, likely have worse quality-of-life features (e.g., wages, benefits, work culture) than similarly impactful roles in scaling labs. People seeking jobs in AI safety might choose between two high-impact roles based on these salient features without considering how many others making the same decisions will affect the talent flow en masse. Programs like MATS might contribute to this problem, but only if the labs keep hiring talent (unlikely given poor returns on scale) and the AI gov/policy orgs don’t make attractive offers (unlikely given METR and Apollo pay pretty good wages, high status, and work cultures comparable to labs; AISIs might be limited because government roles don’t typically pay well, but it seems there are substantial status benefits to working there).
AI risk might be particularly appealing as a cause area to people who are dispositionally and experientially suited to technical work and scaling labs might be the most impactful place to do many varieties of technical work. Programs like MATS are definitely not a detriment here, as they mostly attract individuals who were already going to work in technical careers, expose them to governance-adjacent research like evals, and recommend potential careers in AI gov/policy.
Cheers, Akash! Yep, our confirmed mentor list updated in the days after publishing this retrospective. Our website remains the best up-to-date source for our Summer/Winter plans.
Do you think this is the best thing for MATS to be focusing on, relative to governance/policy?
MATS is not currently bottlenecked on funding for our current Summer plans and hopefully won’t be for Winter either. If further interested high-impact AI gov mentors appear in the next month or two (and some already seem to be appearing), we will boost this component of our Winter research portfolio. If ERA disappeared tomorrow, we would do our best to support many of their AI gov mentors. In my opinion, MATS is currently not sacrificing opportunities to significantly benefit AI governance and policy; rather, we are rate-limited by factors outside of our control and are taking substantial steps to circumvent these, including:
Substantial outreach to potential AI gov mentors;
Pursuing institutional partnerships with key AI gov/policy orgs;
Offering institutional support and advice to other training programs;
Considering alternative program forms less associated with rationality/longtermism;
Connecting scholars and alumni with recommended opportunities in AI gov/policy;
Regularly recommending scholars and alumni to AI gov/policy org hiring managers.
We appreciate further advice to this end!
Do you think there are some cultural things that ought to be examined to figure out why scaling labs are so much more attractive than options that at-least-to-me seem more impactful in expectation?
I think this is a good question, but it might be misleading in isolation. I would additionally ask:
“How many people are the AISIs, METR, and Apollo currently hiring and are they mainly for technical or policy roles? Do we expect this to change?”
“Are the available job opportunities for AI gov researchers and junior policy staffers sufficient to justify pursuing this as a primary career pathway if one is already experienced at ML and particularly well-suited (e.g., dispositionally) for empirical research?”
“Is there a large demand for AI gov researchers with technical experience in AI safety and familiarity with AI threat models, or will most roles go to experienced policy researchers, including those transitioning from other fields? If the former, where should researchers gain technical experience? If the latter, should we be pushing junior AI gov training programs or retraining bootcamps/workshops for experienced professionals?”
“Are existing talent pipelines into AI gov/policy meeting the needs of established research organizations and think tanks (e.g., RAND, GovAI, TFS, IAPS, IFP, etc.)? If not, where can programs like MATS/ERA/etc. best add value?”
“Is there a demand for more organizations like CAIP? If so, what experience do the founders require?”
Thanks for this (very thorough) answer. I’m especially excited to see that you’ve reached out to 25 AI gov researchers & already have four governance mentors for summer 2024. (Minor: I think the post mentioned that you plan to have at least 2, but it seems like there are already 4 confirmed and you’re open to more; apologies if I misread something though.)
A few quick responses to other stuff:
I appreciate a lot of the other content presented. It feels to me like a lot of it is addressing the claim “it is net positive for MATS to upskill people who end up working at scaling labs”, whereas I think the claims I made were a bit different. (Specifically, I think I was going for more “Do you think this is the best thing for MATS to be focusing on, relative to governance/policy”and “Do you think there are some cultural things that ought to be examined to figure out why scaling labs are so much more attractive than options that at-least-to-me seem more impactful in expectation”).
RE AI control, I don’t think I’m necessarily underestimating its popularity as a metastrategy. I’m broadly aware that a large fraction of the Bay Area technical folks are excited about control. However, I think when characterizing the AI safety community as a whole (not just technical people), the shift toward governance/policy macrostrategies is (much) stronger than the shift toward the control macrostrategy. (Separately, I think I’m more excited about foundational work in AI control that looks more like the kind of thing that Buck/Ryan have written about is separate from typical prosaic work (e.g., interpretability), even though lots of typical prosaic work could be argued to be connected to the control macrostrategy.)
+1 that AI governance mentors might be harder to find for some of the reasons you listed.
As a naive guess, I would consider the main reasons to be:
People seeking jobs in AI safety often want to take on “heroic responsibility.” Work on evals and policy, while essential, might be seen as “passing the buck” onto others, often at scaling labs, who have to “solve the wicked problem of AI alignment/control” (quotes indicate my caricature of a hypothetical person). Anecdotally, I’ve often heard people in-community disparage AI safety strategies that primarily “buy time” without “substantially increasing the odds AGI is aligned.” Programs like MATS emphasizing the importance of AI governance and including AI strategy workshops might help shift this mindset, if it exists.
Roles in AI gov/policy, while impactful at reducing AI risk, likely have worse quality-of-life features (e.g., wages, benefits, work culture) than similarly impactful roles in scaling labs. People seeking jobs in AI safety might choose between two high-impact roles based on these salient features without considering how many others making the same decisions will affect the talent flow en masse. Programs like MATS might contribute to this problem, but only if the labs keep hiring talent (unlikely given poor returns on scale) and the AI gov/policy orgs don’t make attractive offers (unlikely given METR and Apollo pay pretty good wages, high status, and work cultures comparable to labs; AISIs might be limited because government roles don’t typically pay well, but it seems there are substantial status benefits to working there).
AI risk might be particularly appealing as a cause area to people who are dispositionally and experientially suited to technical work and scaling labs might be the most impactful place to do many varieties of technical work. Programs like MATS are definitely not a detriment here, as they mostly attract individuals who were already going to work in technical careers, expose them to governance-adjacent research like evals, and recommend potential careers in AI gov/policy.
Cheers, Akash! Yep, our confirmed mentor list updated in the days after publishing this retrospective. Our website remains the best up-to-date source for our Summer/Winter plans.
MATS is not currently bottlenecked on funding for our current Summer plans and hopefully won’t be for Winter either. If further interested high-impact AI gov mentors appear in the next month or two (and some already seem to be appearing), we will boost this component of our Winter research portfolio. If ERA disappeared tomorrow, we would do our best to support many of their AI gov mentors. In my opinion, MATS is currently not sacrificing opportunities to significantly benefit AI governance and policy; rather, we are rate-limited by factors outside of our control and are taking substantial steps to circumvent these, including:
Substantial outreach to potential AI gov mentors;
Pursuing institutional partnerships with key AI gov/policy orgs;
Offering institutional support and advice to other training programs;
Considering alternative program forms less associated with rationality/longtermism;
Connecting scholars and alumni with recommended opportunities in AI gov/policy;
Regularly recommending scholars and alumni to AI gov/policy org hiring managers.
We appreciate further advice to this end!
I think this is a good question, but it might be misleading in isolation. I would additionally ask:
“How many people are the AISIs, METR, and Apollo currently hiring and are they mainly for technical or policy roles? Do we expect this to change?”
“Are the available job opportunities for AI gov researchers and junior policy staffers sufficient to justify pursuing this as a primary career pathway if one is already experienced at ML and particularly well-suited (e.g., dispositionally) for empirical research?”
“Is there a large demand for AI gov researchers with technical experience in AI safety and familiarity with AI threat models, or will most roles go to experienced policy researchers, including those transitioning from other fields? If the former, where should researchers gain technical experience? If the latter, should we be pushing junior AI gov training programs or retraining bootcamps/workshops for experienced professionals?”
“Are existing talent pipelines into AI gov/policy meeting the needs of established research organizations and think tanks (e.g., RAND, GovAI, TFS, IAPS, IFP, etc.)? If not, where can programs like MATS/ERA/etc. best add value?”
“Is there a demand for more organizations like CAIP? If so, what experience do the founders require?”