Give me feedback! :)
Co-Director at ML Alignment & Theory Scholars Program (2022-current)
Co-Founder & Board Member at London Initiative for Safe AI (2023-current)
Manifund Regrantor (2023-current)
Ph.D. in Physics from the University of Queensland (2017-2022)
Group organizer at Effective Altruism UQ (2018-2021)
Also note that historically many individuals entering AI safety seem to have been pursuing the “Connector” path, when most jobs now (and probably in the future) are “Iterator”-shaped, and larger AI safety projects are also principally bottlenecked by “Amplifiers”. The historical focus on recruiting and training Connectors to the detriment of Iterators and Amplifiers has likely contributed to this relative talent shortage. A caveat: Connectors are also critical for founding new research agendas and organizations, though many self-styled Connectors would likely substantially benefit as founders by improving some Amplifier-shaped soft skills, including leadership, collaboration, networking, and fundraising.
In theory, sure! I know @yanni kyriacos recently assessed the need for an ANZ AI safety hub, but I think he concluded there wasn’t enough of a need yet?
@Elizabeth, Mesa nails it above. I would also add that I am conceptualizing impactful AI safety research as the product of multiple reagents, including talent, ideas, infrastructure, and funding. In my bullet point, I was pointing to an abundance of talent and ideas relative to infrastructure and funding. I’m still mostly working on talent development at MATS, but I’m also helping with infrastructure and funding (e.g., founding LISA, advising Catalyze Impact, regranting via Manifund) and I want to do much more for these limiting reagents.
I would amend it to say “sometimes struggles to find meaningful employment despite having the requisite talent to further impactful research directions (which I believe are plentiful)”
Why does the AI safety community need help founding projects?
AI safety should scale
Labs need external auditors for the AI control plan to work
We should pursue many research bets in case superalignment/control fails
Talent leaves MATS/ARENA and sometimes struggles to find meaningful work for mundane reasons, not for lack of talent or ideas
Some emerging research agendas don’t have a home
There are diminishing returns at scale for current AI safety teams; sometimes founding new projects is better than joining an existing team
Scaling lab alignment teams are bottlenecked by management capacity, so their talent cut-off is above the level required to do “useful AIS work”
Research organizations (inc. nonprofits) are often more effective than independent researchers
“Block funding model” is more efficient, as researchers can spend more time researching, rather than seeking grants, managing, or other traditional PI duties that can be outsourced
Open source/collective projects often need a central rallying point (e.g., EleutherAI, dev interp at Timaeus, selection theorems and cyborgism agendas seem too delocalized, etc.)
There is (imminently) a market for for-profit AI safety companies and value-aligned people should capture this free energy or let worse alternatives flourish
If labs or API users are made legally liable for their products, they will seek out external red-teaming/auditing consultants to prove they “made a reasonable attempt” to mitigate harms
If government regulations require labs to seek external auditing, there will be a market for many types of companies
“Ethical AI” companies might seek out interpretability or bias/fairness consultants
New AI safety organizations struggle to get funding and co-founders despite having good ideas
AIS researchers are usually not experienced entrepeneurs (e.g., don’t know how to write grant proposals for EA funders, pitch decks for VCs, manage/hire new team members, etc.)
There are not many competent start-up founders in the EA/AIS community and when they join, they don’t know what is most impactful to help
Creating a centralized resource for entrepeneurial education/consulting and co-founder pairing would solve these problems
AI that obeys the intention of a human user can be asked to help build unsafe AGI, such as by serving as a coding assistant.
I think a better example of your point is “Corrigible AI can be used by a dictator to enforce their rule.”
Yep, it was pointed out to me by @LauraVaughan (and I agree) that e.g. working for RAND or a similar government think tank is another high-impact career pathway in the “Nationalized AGI” future.
Yeah, I basically agree with this nuance. MATS really doesn’t want to overanchor on CodeSignal tests or publication count in scholar selection.
I do think category theory professors or similar would be reasonable advisors for certain types of MIRI research.
Yes to all this, but also I’ll go one level deeper. Even if I had tons more Manifund money to give out (and assuming all the talent needs discussed in the report are saturated with funding), it’s not immediately clear to me that “giving 1-3 year stipends to high-calibre young researchers, no questions asked” is the right play if they don’t have adequate mentorship, the ability to generate useful feedback loops, researcher support systems, access to frontier models if necessary, etc.
I want to sidestep critique of “more exploratory AI safety PhDs” for a moment and ask: why doesn’t MIRI sponsor high-calibre young researchers with a 1-3 year basic stipend and mentorship? And why did MIRI let Vivek’s team go?
We changed the title. I don’t think keeping the previous title was aiding understanding at this point.
I like Adam’s description of an exploratory AI safety PhD:
You’ll also have an unusual degree of autonomy: You’re basically guaranteed funding and a moderately supportive environment for 3-5 years, and if you have a hands-off advisor you can work on pretty much any research topic. This is enough time to try two or more ambitious and risky agendas.
Ex ante funding guarantees, like The Vitalik Buterin PhD Fellowship in AI Existential Safety or Manifund or other funders, mitigate my concerns around overly steering exploratory research. Also, if one is worried about culture/priority drift, there are several AI safety offices in Berkeley, Boston, London, etc. where one could complete their PhD while surrounded by AI safety professionals (which I believe was one of the main benefits of the late Lightcone office).
I am a Manifund Regrantor. In addition to general grantmaking, I have requests for proposals in the following areas:
Funding for AI safety PhDs (e.g., with these supervisors), particularly in exploratory research connecting AI safety theory with empirical ML research.
An AI safety PhD advisory service that helps prospective PhD students choose a supervisor and topic (similar to Effective Thesis, but specialized for AI safety).
Initiatives to critically examine current AI safety macrostrategy (e.g., as articulated by Holden Karnofsky) like the Open Philanthropy AI Worldviews Contest and Future Fund Worldview Prize.
Initiatives to identify and develop “Connectors” outside of academia (e.g., a reboot of the Refine program, well-scoped contests, long-term mentoring and peer-support programs).
Physical community spaces for AI safety in AI hubs outside of the SF Bay Area or London (e.g., Japan, France, Bangalore).
Start-up incubators for projects, including evals/red-teaming/interp companies, that aim to benefit AI safety, like Catalyze Impact, Future of Life Foundation, and YCombinator’s request for Explainable AI start-ups.
Initiatives to develop and publish expert consensus on AI safety macrostrategy cruxes, such as the Existential Persuasion Tournament and 2023 Expert Survey on Progress in AI (e.g., via the Delphi method, interviews, surveys, etc.).
Ethics/prioritization research into:
What values to instill in artificial superintelligence?
How should AI-generated wealth be distributed?
What should people do in a post-labor society?
What level of surveillance/restriction is justified by the Unilateralist’s Curse?
What moral personhood will digital minds have?
How should nations share decision making power regarding transformative AI?
New nonprofit startups that aim to benefit AI safety.
I plan to respond regarding MATS’ future priorities when I’m able (I can’t speak on behalf of MATS alone here and we are currently examining priorities in the lead up to our Winter 2024-25 Program), but in the meantime I’ve added some requests for proposals to my Manifund Regrantor profile.
An interesting note: I don’t necessarily want to start a debate about the merits of academia, but “fund a smart motivated youngster without a plan for 3 years with little evaluation” sounds a lot like “fund more exploratory AI safety PhDs” to me. If anyone wants to do an AI safety PhD (e.g., with these supervisors) and needs funding, I’m happy to evaluate these with my Manifund Regrantor hat on.
I can understand if some people are confused by the title, but we do say “the talent needs of safety teams” in the first sentence. Granted, this doesn’t explicitly reference “funding opportunities” too, but it does make it clear that it is the (unfulfilled) needs of existent safety teams that we are principally referring to.
As a concrete proposal, if anyone wants to reboot Refine or similar, I’d be interested to consider that while wearing my Manifund Regrantor hat.
I interpret your comment as assuming that new researchers with good ideas produce more impact on their own than in teams working towards a shared goal; this seems false to me. I think that independent research is usually a bad bet in general and that most new AI safety researchers should be working on relatively few impactful research directions, most of which are best pursued within a team due to the nature of the research (though some investment in other directions seems good for the portfolio).
I’ve addressed this a bit in thread, but here are some more thoughts:
New AI safety researchers seem to face mundane barriers to reducing AI catastrophic risk, including funding, infrastructure, and general executive function.
MATS alumni are generally doing great stuff (~46% currently work in AI safety/control, ~1.4% work on AI capabilities), but we can do even better.
Like any other nascent scientific/engineering discipline, AI safety will produce more impactful research with scale, albeit with some diminishing returns on impact eventually (I think we are far from the inflection point, however).
MATS alumni, as a large swathe of the most talented new AI safety researchers in my (possibly biased) opinion, should ideally not experience mundane barriers to reducing AI catastrophic risk.
Independent research seems worse than team-based research for most research that aims to reduce AI catastrophic risk:
“Pair-programming”, builder-breaker, rubber-ducking, etc. are valuable parts of the research process and are benefited by working in a team.
Funding insecurity and grantwriting responsibilities are larger for independent researchers and obstruct research.
Orgs with larger teams and discretionary funding can take on interns to help scale projects and provide mentorship.
Good prosaic AI safety research largely looks more like large teams doing engineering and less like lone geniuses doing maths. Obviously, some lone genius researchers (especially on mathsy non-prosaic agendas) seem great for the portfolio too, but these people seem hard to find/train anyways (so there is often more alpha in the former by my lights). Also, I have doubts that the optimal mechanism to incentivize “lone genius research” is via small independent grants instead of large bounties and academic nerdsniping.
Therefore, more infrastructure and funding for MATS alumni, who are generally value-aligned and competent, is good for reducing AI catastrophic risk in expectation.