Give me feedback! :)
Co-Director at ML Alignment & Theory Scholars Program (2022-current)
Co-Founder & Board Member at London Initiative for Safe AI (2023-current)
Manifund Regrantor (2023-current)
Ph.D. in Physics from the University of Queensland (2017-2022)
Group organizer at Effective Altruism UQ (2018-2021)
Wow, high praise for MATS! Thank you so much :) This list is also great for our Summer 2024 Program planning.
Another point: Despite our broad call for mentors, only ~2 individuals expressed interest in mentorship who we did not ultimately decide to support. It’s possible our outreach could be improved and I’m happy to discuss in DMs.
I don’t see this distribution of research projects as “Goodharting” or “overfocusing” on projects with clear feedback loops. As MATS is principally a program for prosaic AI alignment at the moment, most research conducted within the program should be within this paradigm. We believe projects that frequently “touch reality” often offer the highest expected value in terms of reducing AI catastrophic risk, and principally support non-prosaic, “speculative,” and emerging research agendas for their “exploration value,” which might aid potential paradigm shifts, as well as to round out our portfolio (i.e., “hedge our bets”).
However, even with the focus on prosaic AI alignment research agendas, our Summer 2023 Program supported many emerging or neglected research agendas, including projects in agent foundations, simulator theory, cooperative/multipolar AI (including s-risks), the nascent “activation engineering” approach our program helped pioneer, and the emerging “cyborgism” research agenda.
Additionally, our mentor portfolio is somewhat conditioned on the preferences of our funders. While we largely endorse our funders’ priorities, we are seeking additional funding diversification so that we can support further speculative “research bets”. If you are aware of large funders willing to support our program, please let me know!
There seems to be a bit of pushback against “postmortem” and our team is ambivalent, so I changed to “retrospective.”
Thank you!
Ok, added!
FYI, the Net Promoter score is 38%.
Ok, graph is updated!
Do you think “46% of scholar projects were rated 9⁄10 or higher” is better? What about “scholar projects were rated 8.1/10 on average” ?
We also asked mentors to rate scholars’ “depth of technical ability,” “breadth of AI safety knowledge,” “research taste,” and “value alignment.” We ommitted these results from the report to prevent bloat, but your comment makes me think we should re-add them.
Yeah, I just realized the graph is wrong; it seems like the 10⁄10 scores were truncated. We’ll upload a new graph shortly.
Cheers, Vaniver! As indicated in the figure legend for “Mentor ratings of scholar research”, mentors were asked, “Taking the above [depth/breadth/taste ratings] into account, how strongly do you support the scholar’s research continuing?” and prompted with:
10⁄10 = Very disappointed if [the research] didn’t continue;
5⁄10 = On the fence, unsure what the right call is;
1⁄10 = Fine if research doesn’t continue.
Mentors rated 18% of scholar research projects as 10⁄10 and 28% as 9⁄10.
Also, last year’s program was 8-weeks and this year’s program is 10-weeks.
Buck Shlegeris, Ethan Perez, Evan Hubinger, and Owain Evans are mentoring in both programs. The links show their MATS projects, “personal fit” for applicants, and (where applicable) applicant selection questions, designed to mimic the research experience.
Astra seems like an obviously better choice for applicants principally interested in:
AI governance: MATS has no AI governance mentors in the Winter 2023-24 Program, whereas Astra has Daniel Kokotajlo, Richard Ngo, and associated staff at ARC Evals and Open Phil;
Worldview investigations: Astra has Ajeya Cotra, Tom Davidson, and Lukas Finnvedan, whereas MATS has no Open Phil mentors;
ARC Evals: While both programs feature mentors working on evals, only Astra is working with ARC Evals;
AI ethics: Astra is working with Rob Long.
MATS has the following features that might be worth considering:
Empowerment: Emphasis on empowering scholars to develop as future “research leads” (think accelerated PhD-style program rather than a traditional internship), including research strategy workshops, significant opportunities for scholar project ownership (though the extent of this varies between mentors), and a 4-month extension program;
Diversity: Emphasis on a broad portfolio of AI safety research agendas and perspectives with a large, diverse cohort (50-60) and comprehensive seminar program;
Support: Dedicated and experienced scholar support + research coach/manager staff and infrastructure;
Network: Large and supportive alumni network that regularly sparks research collaborations and AI safety start-ups (e.g., Apollo, Leap Labs, Timaeus, Cadenza, CAIP);
Experience: Have run successful research cohorts with 30, 58, 60 scholars, plus three extension programs with about half as many participants.
Our alumni were surveyed about what amount of funding they would consider sufficient to participate in MATS and the median (discounting the $0 responses) was quite low.
I found this article useful. Any plans to update this for 2024?