Thank you so much for conducting this survey! I want to share some information on behalf of MATS:
In comparison to the AIS survey gender ratio of 9 M:F, MATS Winter 2023-24 scholars and mentors were 4 M:F and 12 M:F, respectively. Our Winter 2023-24 applicants were 4.6 M:F, whereas our Summer 2024 applicants were 2.6 M:F, closer to the EA survey ratio of 2 M:F. This data seems to indicate a large recent change in gender ratios of people entering the AIS field. Did you find that your AIS survey respondents with more AIS experience were significantly more male than newer entrants to the field?
MATS Summer 2024 applicants and interested mentors similarly prioritized research to “understand existing models”, such as interpretability and evaluations, over research to “control the AI” or “make the AI solve it”, such as scalable oversight and control/red-teaming, over “theory work”, such as agent foundations and cooperative AI (note that some cooperative AI work is primarily empirical).
The forthcoming summary of our “AI safety talent needs” interview series generally agrees with this survey’s findings regarding the importance of “soft skills” and “work ethic” in impactful new AIS contributors. Watch this space!
In addition to supporting core established AIS research paradigms, MATS would like to encourage the development of new paradigms. For better or worse, the current AIS funding landscape seems to have a high bar for speculative research into new paradigms. Has AE Studios considered sponsoring significant bounties or impact markets for scoping promising new AIS research directions?
Did survey respondents mention how they proposed making AIS more multidisciplinary? Which established research fields are more needed in the AIS community?
Did EAs consider AIS exclusively a longtermist cause area, or did they anticipate near-term catastrophic risk from AGI?
Thank you for the kind donation to MATS as a result of this survey!
Thanks for all these additional datapoints! I’ll try to respond all of your questions in turn:
Did you find that your AIS survey respondents with more AIS experience were significantly more male than newer entrants to the field?
Overall, there don’t appear to be major differences when filtering for amount of alignment experience. When filtering for greater than vs. less than 6 months of experience, it does appear that the ratio looks more like ~5 M:F; at greater than vs. less than 1 year of experience, it looks like ~8 M:F; the others still look like ~9 M:F. Perhaps the changes you see over the past two years at MATS are too recent to be reflected fully in this data, but it does seem like a generally positive signal that you see this ratio changing (given what we discuss in the post).
Has AE Studio considered sponsoring significant bounties or impact markets for scoping promising new AIS research directions?
We definitely want to do everything we can to support increased exploration of neglected approaches—if you have specific ideas here, we’d love to hear them and discuss more! Maybe we can follow up offline on this.
Did survey respondents mention how they proposed making AIS more multidisciplinary? Which established research fields are more needed in the AIS community?
We don’t appear to have gotten many practical proposals for how to make AIS more multidisciplinary, but there were a number of specific disciplines mentioned in the free responses, including cognitive psychology, neuroscience, game theory, behavioral science, ethics/law/sociology, and philosophy (epistemology was specifically brought up across multiple respondents). One respondent wrote, “AI alignment is dominated by computer scientists who don’t know much about human nature, and could benefit from more behavioral science expertise and game theory,” which I think captures the sentiment of many of the related responses most succinctly (however accurate this statement actually is!). Ultimately, encouraging and funding research at the intersection of these underexplored areas and alignment is likely the only thing that will actually lead to a more multidisciplinary research environment.
Did EAs consider AIS exclusively a longtermist cause area, or did they anticipate near-term catastrophic risk from AGI?
Unfortunately, I don’t think we asked the EA sample about AIS in a way that would allow us to answer this question using the data we have. This would be a really interesting follow-up direction. I will paste in below the ground truth distribution of EAs’ views on the relative promise of these approaches as additional context (eg, we see that the ‘AI risk’ and ‘Existential risk (general)’ distributions have very similar shapes), but I don’t think we can confidently say much about whether these risks were being conceptualized as short- or long-term.
It’s also important to highlight that in the alignment sample (from the other survey), researchers generally indicate that they do not think we’re going to get AGI in the next five years. Again, this doesn’t clarify if they think there are x-risks that could emerge in the nearer term from less-general-but-still-very-advanced AI, but it does provide an additional datapoint that if we are considering AI x-risks to be largely mediated by the advent of AGI, alignment researchers don’t seem to expect this as a whole in the very short term:
You might be interested in this breakdown of gender differences in the research interests of the 719 applicants to the MATS Summer 2024 and Winter 2024-25 Programs who shared their gender. The plot shows the difference between the percentage of male applicants who indicated interest in specific research directions from the percentage of female applicants who indicated interest in the same.
The most male-dominated research interest is mech interp, possibly due to the high male representation in software engineering (~80%), physics (~80%), and mathematics (~60%). The most female-dominated research interest is AI governance, possibly due to the high female representation in the humanities (~60%). Interestingly, cooperative AI was a female-dominated research interest, which seems to match the result from your survey where female respondents were less in favor of “controlling” AIs relative to men and more in favor of “coexistence” with AIs.
Thank you so much for conducting this survey! I want to share some information on behalf of MATS:
In comparison to the AIS survey gender ratio of 9 M:F, MATS Winter 2023-24 scholars and mentors were 4 M:F and 12 M:F, respectively. Our Winter 2023-24 applicants were 4.6 M:F, whereas our Summer 2024 applicants were 2.6 M:F, closer to the EA survey ratio of 2 M:F. This data seems to indicate a large recent change in gender ratios of people entering the AIS field. Did you find that your AIS survey respondents with more AIS experience were significantly more male than newer entrants to the field?
MATS Summer 2024 applicants and interested mentors similarly prioritized research to “understand existing models”, such as interpretability and evaluations, over research to “control the AI” or “make the AI solve it”, such as scalable oversight and control/red-teaming, over “theory work”, such as agent foundations and cooperative AI (note that some cooperative AI work is primarily empirical).
The forthcoming summary of our “AI safety talent needs” interview series generally agrees with this survey’s findings regarding the importance of “soft skills” and “work ethic” in impactful new AIS contributors. Watch this space!
In addition to supporting core established AIS research paradigms, MATS would like to encourage the development of new paradigms. For better or worse, the current AIS funding landscape seems to have a high bar for speculative research into new paradigms. Has AE Studios considered sponsoring significant bounties or impact markets for scoping promising new AIS research directions?
Did survey respondents mention how they proposed making AIS more multidisciplinary? Which established research fields are more needed in the AIS community?
Did EAs consider AIS exclusively a longtermist cause area, or did they anticipate near-term catastrophic risk from AGI?
Thank you for the kind donation to MATS as a result of this survey!
Thanks for all these additional datapoints! I’ll try to respond all of your questions in turn:
Overall, there don’t appear to be major differences when filtering for amount of alignment experience. When filtering for greater than vs. less than 6 months of experience, it does appear that the ratio looks more like ~5 M:F; at greater than vs. less than 1 year of experience, it looks like ~8 M:F; the others still look like ~9 M:F. Perhaps the changes you see over the past two years at MATS are too recent to be reflected fully in this data, but it does seem like a generally positive signal that you see this ratio changing (given what we discuss in the post).
We definitely want to do everything we can to support increased exploration of neglected approaches—if you have specific ideas here, we’d love to hear them and discuss more! Maybe we can follow up offline on this.
We don’t appear to have gotten many practical proposals for how to make AIS more multidisciplinary, but there were a number of specific disciplines mentioned in the free responses, including cognitive psychology, neuroscience, game theory, behavioral science, ethics/law/sociology, and philosophy (epistemology was specifically brought up across multiple respondents). One respondent wrote, “AI alignment is dominated by computer scientists who don’t know much about human nature, and could benefit from more behavioral science expertise and game theory,” which I think captures the sentiment of many of the related responses most succinctly (however accurate this statement actually is!). Ultimately, encouraging and funding research at the intersection of these underexplored areas and alignment is likely the only thing that will actually lead to a more multidisciplinary research environment.
Unfortunately, I don’t think we asked the EA sample about AIS in a way that would allow us to answer this question using the data we have. This would be a really interesting follow-up direction. I will paste in below the ground truth distribution of EAs’ views on the relative promise of these approaches as additional context (eg, we see that the ‘AI risk’ and ‘Existential risk (general)’ distributions have very similar shapes), but I don’t think we can confidently say much about whether these risks were being conceptualized as short- or long-term.
It’s also important to highlight that in the alignment sample (from the other survey), researchers generally indicate that they do not think we’re going to get AGI in the next five years. Again, this doesn’t clarify if they think there are x-risks that could emerge in the nearer term from less-general-but-still-very-advanced AI, but it does provide an additional datapoint that if we are considering AI x-risks to be largely mediated by the advent of AGI, alignment researchers don’t seem to expect this as a whole in the very short term:
You might be interested in this breakdown of gender differences in the research interests of the 719 applicants to the MATS Summer 2024 and Winter 2024-25 Programs who shared their gender. The plot shows the difference between the percentage of male applicants who indicated interest in specific research directions from the percentage of female applicants who indicated interest in the same.
The most male-dominated research interest is mech interp, possibly due to the high male representation in software engineering (~80%), physics (~80%), and mathematics (~60%). The most female-dominated research interest is AI governance, possibly due to the high female representation in the humanities (~60%). Interestingly, cooperative AI was a female-dominated research interest, which seems to match the result from your survey where female respondents were less in favor of “controlling” AIs relative to men and more in favor of “coexistence” with AIs.