Give me feedback! :)
Co-Director at ML Alignment & Theory Scholars Program (2022-current)
Co-Founder & Board Member at London Initiative for Safe AI (2023-current)
Manifund Regrantor (2023-current)
Ph.D. in Physics from the University of Queensland (2017-2022)
Group organizer at Effective Altruism UQ (2018-2021)
Thank you so much for conducting this survey! I want to share some information on behalf of MATS:
In comparison to the AIS survey gender ratio of 9 M:F, MATS Winter 2023-24 scholars and mentors were 4 M:F and 12 M:F, respectively. Our Winter 2023-24 applicants were 4.6 M:F, whereas our Summer 2024 applicants were 2.6 M:F, closer to the EA survey ratio of 2 M:F. This data seems to indicate a large recent change in gender ratios of people entering the AIS field. Did you find that your AIS survey respondents with more AIS experience were significantly more male than newer entrants to the field?
MATS Summer 2024 applicants and interested mentors similarly prioritized research to “understand existing models”, such as interpretability and evaluations, over research to “control the AI” or “make the AI solve it”, such as scalable oversight and control/red-teaming, over “theory work”, such as agent foundations and cooperative AI (note that some cooperative AI work is primarily empirical).
The forthcoming summary of our “AI safety talent needs” interview series generally agrees with this survey’s findings regarding the importance of “soft skills” and “work ethic” in impactful new AIS contributors. Watch this space!
In addition to supporting core established AIS research paradigms, MATS would like to encourage the development of new paradigms. For better or worse, the current AIS funding landscape seems to have a high bar for speculative research into new paradigms. Has AE Studios considered sponsoring significant bounties or impact markets for scoping promising new AIS research directions?
Did survey respondents mention how they proposed making AIS more multidisciplinary? Which established research fields are more needed in the AIS community?
Did EAs consider AIS exclusively a longtermist cause area, or did they anticipate near-term catastrophic risk from AGI?
Thank you for the kind donation to MATS as a result of this survey!
I’m skeptical of the usefulness of such an organization. I think we have a plethora of motivated talent passing through AGISF currently that doesn’t need another short workshop or a bunch of low-context 1-1s with researchers who probably have better-vetted people to spend their scarce time on (I think vetting is very hard). I think the AI safety talent development pipeline needs less centralized, short-duration, broad-funnel workshops and more programs that allow for longer-term talent incubation into specific niches (e.g., programs/low-stakes jobs that build critical skills in research vs. management vs. operations) that don’t eat up valuable researcher time unnecessarily and encourage decentralization of alignment research hubs. Sorry if this sounds like bad-faith criticism; it’s not intended to be.
Seems like a great idea. I think I’d strictly prefer “a combined research mentorship and seminar program that aims to do for AI governance research what MATS is trying to do for technical AI alignment research,” because it feels like the optimal program for Cotra or Kokotajlo is a bit different from MATS and likely includes other great governance/worldview/macrostrategy researchers. However, I’ll probably talk to Cotra and Kokotajlo to see if we can add value to them!
I think this is rather the domain of existing academic and industry research groups (e.g., Krueger’s lab, Anthropic, CHAI, OpenAI safety team, DeepMind safety team, etc.), as these groups have the necessary talent and, presumably, motivation. I’d also be excited for MATS scholars and alumni and LTFF-funded independent researchers to work on this.
Seems good, if hard (and not something I’d expect a competition to help with, on priors, if the most capable/aligned people already are not working on this). In particular, I’d be excited to see something that discusses, from first principles, whether solving alignment is significantly harder than similar scientific problems (credit to Caleb for often talking about this).
I’m skeptical that this would be low-risk (in regards to making researchers more skeptical of alignment and less likely to listen to AI safety spokespeople at a critical date) or a counterfactually valuable use of senior AI safety researcher time.
Last program, 44% of scholar research was on interpretability, 18% on evals/demos, 17% on oversight/control, etc. In summer, we intend for 35% of scholar research to be on interpretability, 17% on evals/demos, 27% on oversight/control, etc., based on our available mentor pool and research priorities. Interpretability will still be the largest research track and still has the greatest interest from potential mentors and applicants. The plot below shows the research interests of 1331 MATS applicants and 54 potential mentors who have applied for our Summer 2024 or Winter 2024-25 Programs.
I think the high interest in working at scaling labs relative to governance or nonprofit organizations can be explained by:
Most of the scholars in this cohort were working on research agendas for which there are world-leading teams based at scaling labs (e.g., 44% interpretability, 17% oversight/control). Fewer total scholars were working on evals/demos (18%), agent foundations (8%), and formal verification (3%). Therefore, I would not be surprised if many scholars wanted to pursue interpretability or oversight/control at scaling labs.
There seems to be an increasing trend in the AI safety community towards the belief that most useful alignment research will occur at scaling labs (particularly once there are automated research assistants) and external auditors with privileged frontier model access (e.g., METR, Apollo, AISIs). This view seems particularly strongly held by proponents of the “AI control” metastrategy.
Anecdotally, scholars seemed generally in favor of careers at an AISI or evals org, but would prefer to continue pursuing their current research agenda (which might be overdetermined given the large selection pressure they faced to get into MATS to work on that agenda).
Starting new technical AI safety orgs/projects seems quite difficult in the current funding ecosystem. I know of many alumni who have founded or are trying to found projects who express substantial difficulties with securing sufficient funding.
Note that the career fair survey might tell us little about how likely scholars are to start new projects as it was primarily seeking interest in which organizations should attend, not in whether scholars should join orgs vs. found their own.
I don’t see this distribution of research projects as “Goodharting” or “overfocusing” on projects with clear feedback loops. As MATS is principally a program for prosaic AI alignment at the moment, most research conducted within the program should be within this paradigm. We believe projects that frequently “touch reality” often offer the highest expected value in terms of reducing AI catastrophic risk, and principally support non-prosaic, “speculative,” and emerging research agendas for their “exploration value,” which might aid potential paradigm shifts, as well as to round out our portfolio (i.e., “hedge our bets”).
However, even with the focus on prosaic AI alignment research agendas, our Summer 2023 Program supported many emerging or neglected research agendas, including projects in agent foundations, simulator theory, cooperative/multipolar AI (including s-risks), the nascent “activation engineering” approach our program helped pioneer, and the emerging “cyborgism” research agenda.
Additionally, our mentor portfolio is somewhat conditioned on the preferences of our funders. While we largely endorse our funders’ priorities, we are seeking additional funding diversification so that we can support further speculative “research bets”. If you are aware of large funders willing to support our program, please let me know!
An incomplete list of possibly useful AI safety research:
Predicting/shaping emergent systems (“physics”)
Learning theory (e.g., shard theory, causal incentives)
Regularizers (e.g., speed priors)
Embedded agency (e.g., infra-Bayesianism, finite factored sets)
Decision theory (e.g., timeless decision theory, cooperative bargaining theory, acausal trade)
Model evaluation (“biology”)
Capabilities evaluation (e.g., survive-and-spread, hacking)
Red-teaming alignment techniques
Demonstrations of emergent properties/behavior (e.g., instrumental powerseeking)
Interpretability (“neuroscience”)
Mechanistic interpretability (e.g., superposition, toy models, automated circuit detection)
Gray box ELK (e.g., Collin Burns’ research)
Feature extraction/sparsity (including Wentworth/Bushnaq style “modularity” research)
Model surgery (e.g., ROME)
Alignment MVP (“psychology”)
Sampling simulators safely (conditioning predictive models)
Scalable oversight (e.g., RLHF, CAI, debate, RRM, model-assisted evaluations)
Cyborgism
Prompt engineering (e.g., jailbreaking)
Strategy/governance (“sociology”)
Compute governance (e.g., GPU logging/restrictions, treaties)
Model safety standards (e.g., auditing policies)
Infosecurity
Multi-party authentication
Airgapping
AI-assisted infosecurity
Of the scholars ranked 5⁄10 and lower on value alignment, 63% worked with a mentor at a scaling lab, compared with 27% of the scholars ranked 6⁄10 and higher. The average scaling lab mentors rated their scholars’ value alignment at 7.3/10 and rated 78% of their scholars at 6⁄10 and higher, compared to 8.0/10 and 90% for the average non-scaling lab mentor. This indicates that our scaling lab mentors were more discerning of value alignment on average than non-scaling lab mentors, or had a higher base rate of low-value alignment scholars (probably both).
I also want to push back a bit against an implicit framing of the average scaling lab safety researcher we support as being relatively unconcerned about value alignment or the positive impact of their research; this seems manifestly false from my conversations with mentors, their scholars, and the broader community.
Our alumni were surveyed about what amount of funding they would consider sufficient to participate in MATS and the median (discounting the $0 responses) was quite low.
AI alignment threat models that are somewhat MECE (but not quite):
We get what we measure (models converge to the human++ simulator and build a Potemkin village world without being deceptive consequentialists);
Optimization daemon (deceptive consequentialist with a non-myopic utility function arises in training and does gradient hacking, buries trojans and obfuscates cognition to circumvent interpretability tools, “unboxes” itself, executes a “treacherous turn” when deployed, coordinates with auditors and future instances of itself, etc.);
Coordination failure (otherwise-aligned AI systems combust or gridlock far from the Pareto frontier due to opaque values/capabilities, inadequate commitment mechanisms, or irreconcilable differences);
Sharp left turn (models learn generally powerful cognitive tools that are efficiently reached by training on real-world tasks, especially as the real world contains useful embedded knowledge that shortcuts learning these tools from scratch; but powerful cognitive tools are somewhat anti-natural to corrigibility and the training process does not efficiently constrain the directionality of these tools towards human CEV, which manifests under distributional shift).
In particular, the last threat model feels like it is trying to cut across aspects of the first two threat models, violating MECE.
It seems plausible to me that at least some MATS scholars are somewhat motivated by a desire to work at scaling labs for money, status, etc. However, the value alignment of scholars towards principally reducing AI risk seems generally very high. In Winter 2023-24, our most empirical research dominated cohort, mentors rated the median scholar’s value alignment at 8⁄10 and 85% of scholars were rated 6⁄10 or above, where 5⁄10 was “Motivated in part, but would potentially switch focus entirely if it became too personally inconvenient.” To me this is a very encouraging statistic, but I’m sympathetic to concerns that well-intentioned young researchers who join scaling labs might experience value drift, or find it difficult to promote safety culture internally or sound the alarm if necessary; we are consequently planning a “lab safety culture” workshop in Summer. Notably, only 3.7% of surveyed MATS alumni say they are working on AI capabilities; in one case, an alumnus joined a scaling lab capabilties team and transferred to working on safety projects as soon as they were able. As with all things, maximizing our impact is about striking the right balance between trust and caution and I’m encouraged by the high apparent value alignment of our alumni and scholars.
We additionally believe:
Advancing researchers to get hired at lab safety teams is generally good;
We would prefer that the people on lab safety teams have more research experience and are more value-aligned, all else equal, and we think MATS improves scholars on these dimensions;
We would prefer lab safety teams to be larger, and it seems likely that MATS helps create a stronger applicant pool for these jobs, resulting in more hires overall;
MATS creates a pipeline for senior researchers on safety teams to hire people they have worked with for up to 6.5 months in-program, observing their compentency and value alignment;
Even if MATS alumni defect to work on pure capabilities, we would still prefer them to be more value-aligned than otherwise (though of course this has to be weighed against the boost MATS gave to their research abilities).
Regarding “AI control,” I suspect you might be underestimating the support that this metastrategy has garnered in the technical AI safety community, particularly among prosaic AGI safety thought leaders. I see Paul’s decision to leave ARC in favor of the US AISI as a potential endorsement of the AI control paradigm over intent alignment, rather than necessarily an endorsement of an immediate AI pause (I would update against this if he pushes more for a pause than for evals and regulations). I do not support AI control to the exclusion of other metastrategies (including intent alignment and Pause AI), but I consider it a vital and growing component of my strategy portfolio.
It’s true that many AI safety projects are pivoting towards AI governance. I think the establishment of AISIs is wonderful; I am in contact with MATS alumni Alan Cooney and Max Kauffman at the UK AISI and similarly want to help the US AISI with hiring. I would have been excited for Vivek Hebbar’s, Jeremy Gillen’s, Peter Barnett’s, James Lucassen’s, and Thomas Kwa’s research in empirical agent foundations to continue at MIRI, but I am also excited about the new technical governance focus that MATS alumni Lisa Thiergart and Peter Barnett are exploring. I additionally have supported AI safety org accelerator Catalyze Impact as an advisor and Manifund Regrantor and advised several MATS alumni founding AI safety projects; it’s not easy to attract or train good founders!
MATS has been interested in supporting more AI governance research since Winter 2022-23, when we supported Richard Ngo and Daniel Kokotajlo (although both declined to accept scholars past the training program) and offered support to several more AI gov researchers. In Summer 2023, we reached out to seven handpicked governance/strategy mentors (some of which you recommended, Akash), though only one was interested in mentoring. In Winter 2023-24 we tried again, with little success. In preparation for the upcoming Summer 2024 and Winter 2024-25 Programs, we reached out to 25 AI gov/policy/natsec researchers (who we asked to also share with their networks) and received expressions of interest from 7 further AI gov researchers. As you can see from our website, MATS is supporting four AI gov mentors in Summer 2024 (six if you count Matija Franklin and Philip Moreira Tomei, who are primarily working on value alignment). We’ve additionally reached out to RAND, IAPS, and others to provide general support. MATS is considering a larger pivot, but available mentors are clearly a limiting constraint. Please contact me if you’re an AI gov researcher and want to mentor!
Part of the reason that AI gov mentors are harder to find is that programs like the RAND TASP, GovAI, IAPS, Horizon, ERA, etc. fellowships seem to be doing a great job collectively of leveraging the available talent. It’s also possible that AI gov researchers are discouraged from mentoring at MATS because of our obvious associations with AI alignment (it’s in the name) and the Berkeley longtermist/rationalist scene (we’re talking on LessWrong and operate in Berkeley). We are currently considering ways to support AI gov researchers who don’t want to affiliate with the alignment, x-risk, longtermist, or rationalist communities.
I’ll additionally note that MATS has historically supported much research that indirectly contributes to AI gov/policy, such as Owain Evans’, Beth Barnes’, and Francis Rhys Ward’s capabilities evals, Evan Hubinger’s alignment evals, Jeffrey Ladish’s capabilities demos, Jesse Clifton’s and Caspar Oesterheldt’s cooperation mechanisms, etc.
Reasons that scaling labs might be motivated to sign onto AI safety standards:
Companies who are wary of being sued for unsafe deployment that causes harm might want to be able to prove that they credibly did their best to prevent harm.
Big tech companies like Google might not want to risk premature deployment, but might feel forced to if smaller companies with less to lose undercut their “search” market. Standards that prevent unsafe deployment fix this.
However, AI companies that don’t believe in AGI x-risk might tolerate higher x-risk than ideal safety standards by the lights of this community. Also, I think insurance contracts are unlikely to appropriately account for x-risk, if the market is anything to go by.
MATS aims to find and accelerate alignment research talent, including:
Developing scholar research ability through curriculum elements focused on breadth, depth, and originality (the “T-model of research”);
Assisting scholars in producing impactful research through research mentorship, a community of collaborative peers, dedicated 1-1 support, and educational seminars;
Aiding the creation of impactful new alignment organizations (e.g., Jessica Rumbelow’s Leap Labs and Marius Hobbhahn’s Apollo Research);
Preparing scholars for impactful alignment research roles in existing organizations.
Not all alumni will end up in existing alignment research organizations immediately; some return to academia, pursue independent research, or potentially skill-up in industry (to eventually aid alignment research efforts). We generally aim to find talent with existing research ability and empower it to work on alignment, not necessarily through existing initiatives (though we certainly endorse many).
I think there should be much more seminars about philosophy of science, methodology of science, and strategy of research in the program. Perhaps, not at the expense of other seminar content, but via increasing the number of seminar hours. Talks about ethics, and in particular about ethical naturalism (such as Oliver Scott Curry’s “Morality as Cooperation”), or interaction of ethics and science more generally, also seem essential.
MATS is currently focused on developing scholars as per the “T-model of research” with three main levers:
Mentorship (weekly meetings + mentor-specific curriculum);
Curriculum (seminars + workshops + Alignment 201 + topic study groups);
Scholar support (1-1s for unblocking + research strategy).
The “T-model of research” is:
Breadth-first search (literature reviews, building a “toolbox” of knowledge, noticing gaps);
Depth-first search (forming testable hypotheses, executing research, recursing appropriately, using checkpoints);
Originality (identifying threat models, backchaining to local search, applying builder/breaker methodology, babble and prune, “infinite-compute/time” style problem decompositions, etc.).
In the Winter 2022-23 Cohort, we ran several research strategy workshops (focusing on problem decomposition and strategy identification) and had dedicated scholar support staff who offered regular, airgapped 1-1 support for research strategy and researcher unblocking. We will publish some findings from our Scholar Support Post-Mortem soon. We plan to run further research strategy and “originality” workshops in the Summer 2023 Cohort and are currently reviewing our curriculum from the ground up.
Currently, we are not focused on “ethical naturalism” or similar as a curriculum element as it does not seem broadly useful for our cohort compared to project-specific, scholar-driven self-education. We are potentially open to hosting a seminar on this topic.
Additional reasons why MATS + MAGS > MATS + governance patch:
MATS is quite large (55 in-person scholars for winter);
There are good alignment mentors MATS could support that might get bumped out if we add governance mentors;
At some point, more MATS team members (to support a larger cohort) might make it hard for us to work effectively (as per Lightcone’s model);
A small, cohesive MAGS team could work in parallel with the MATS team for greater impact than a unified team;
GovAI are already doing something like the hypothetical MAGS, and the OpenAI governance team might want to do the same, which would mean a lot of potential mentors (maybe enough for a MATS-sized program, funders + mentors willing).
Do you think there are some cultural things that ought to be examined to figure out why scaling labs are so much more attractive than options that at-least-to-me seem more impactful in expectation?
As a naive guess, I would consider the main reasons to be:
People seeking jobs in AI safety often want to take on “heroic responsibility.” Work on evals and policy, while essential, might be seen as “passing the buck” onto others, often at scaling labs, who have to “solve the wicked problem of AI alignment/control” (quotes indicate my caricature of a hypothetical person). Anecdotally, I’ve often heard people in-community disparage AI safety strategies that primarily “buy time” without “substantially increasing the odds AGI is aligned.” Programs like MATS emphasizing the importance of AI governance and including AI strategy workshops might help shift this mindset, if it exists.
Roles in AI gov/policy, while impactful at reducing AI risk, likely have worse quality-of-life features (e.g., wages, benefits, work culture) than similarly impactful roles in scaling labs. People seeking jobs in AI safety might choose between two high-impact roles based on these salient features without considering how many others making the same decisions will affect the talent flow en masse. Programs like MATS might contribute to this problem, but only if the labs keep hiring talent (unlikely given poor returns on scale) and the AI gov/policy orgs don’t make attractive offers (unlikely given METR and Apollo pay pretty good wages, high status, and work cultures comparable to labs; AISIs might be limited because government roles don’t typically pay well, but it seems there are substantial status benefits to working there).
AI risk might be particularly appealing as a cause area to people who are dispositionally and experientially suited to technical work and scaling labs might be the most impactful place to do many varieties of technical work. Programs like MATS are definitely not a detriment here, as they mostly attract individuals who were already going to work in technical careers, expose them to governance-adjacent research like evals, and recommend potential careers in AI gov/policy.
You might be interested in this breakdown of gender differences in the research interests of the 719 applicants to the MATS Summer 2024 and Winter 2024-25 Programs who shared their gender. The plot shows the difference between the percentage of male applicants who indicated interest in specific research directions from the percentage of female applicants who indicated interest in the same.
The most male-dominated research interest is mech interp, possibly due to the high male representation in software engineering (~80%), physics (~80%), and mathematics (~60%). The most female-dominated research interest is AI governance, possibly due to the high female representation in the humanities (~60%). Interestingly, cooperative AI was a female-dominated research interest, which seems to match the result from your survey where female respondents were less in favor of “controlling” AIs relative to men and more in favor of “coexistence” with AIs.
Buck Shlegeris, Ethan Perez, Evan Hubinger, and Owain Evans are mentoring in both programs. The links show their MATS projects, “personal fit” for applicants, and (where applicable) applicant selection questions, designed to mimic the research experience.
Astra seems like an obviously better choice for applicants principally interested in:
AI governance: MATS has no AI governance mentors in the Winter 2023-24 Program, whereas Astra has Daniel Kokotajlo, Richard Ngo, and associated staff at ARC Evals and Open Phil;
Worldview investigations: Astra has Ajeya Cotra, Tom Davidson, and Lukas Finnvedan, whereas MATS has no Open Phil mentors;
ARC Evals: While both programs feature mentors working on evals, only Astra is working with ARC Evals;
AI ethics: Astra is working with Rob Long.
MATS has the following features that might be worth considering:
Empowerment: Emphasis on empowering scholars to develop as future “research leads” (think accelerated PhD-style program rather than a traditional internship), including research strategy workshops, significant opportunities for scholar project ownership (though the extent of this varies between mentors), and a 4-month extension program;
Diversity: Emphasis on a broad portfolio of AI safety research agendas and perspectives with a large, diverse cohort (50-60) and comprehensive seminar program;
Support: Dedicated and experienced scholar support + research coach/manager staff and infrastructure;
Network: Large and supportive alumni network that regularly sparks research collaborations and AI safety start-ups (e.g., Apollo, Leap Labs, Timaeus, Cadenza, CAIP);
Experience: Have run successful research cohorts with 30, 58, 60 scholars, plus three extension programs with about half as many participants.
Yes
Main takeaways from a recent AI safety conference:
If your foundation model is one small amount of RL away from being dangerous and someone can steal your model weights, fancy alignment techniques don’t matter. Scaling labs cannot currently prevent state actors from hacking their systems and stealing their stuff. Infosecurity is important to alignment.
Scaling labs might have some incentive to go along with the development of safety standards as it prevents smaller players from undercutting their business model and provides a credible defense against lawsuits regarding unexpected side effects of deployment (especially with how many tech restrictions the EU seems to pump out). Once the foot is in the door, more useful safety standards to prevent x-risk might be possible.
Near-term commercial AI systems that can be jailbroken to elicit dangerous output might empower more bad actors to make bioweapons or cyberweapons. Preventing the misuse of near-term commercial AI systems or slowing down their deployment seems important.
When a skill is hard to teach, like making accurate predictions over long time horizons in complicated situations or developing a “security mindset,” try treating humans like RL agents. For example, Ph.D. students might only get ~3 data points on how to evaluate a research proposal ex-ante, whereas Professors might have ~50. Novice Ph.D. students could be trained to predict good research decisions by predicting outcomes on a set of expert-annotated examples of research quandaries and then receiving “RL updates” based on what the expert did and what occurred.