Ryan Kidd

• Ryan Kidd 25 May 2026 16:32 UTC

  5 points

  0

  in reply to: Mitchell_Porter’s comment on: Olivia Schar­f­man’s Shortform

  The justification for a philanthropic overhang is:

  • Existing large funders 2-3xing their annual budgets this year;

  • The pledged commitments from Anthropic founders and employee DAFs.;

  • The posts like these from cG staff;

  • The OpenAI Foundation pledging $1B in 2026;

  • Nan Ransohoff’s calculation that “AI will likely add ~$37-100B per year in intended philanthropic spend in the near future.”

• Ryan Kidd 18 May 2026 20:34 UTC

  3 points

  0

  in reply to: Ryan Kidd’s comment on: What holds AI safety to­gether? Co-au­thor­ship net­works from 200 papers

  Also note this database.

MATS Au­tumn 2026 Fel­low­ship Ap­pli­ca­tions Now Open—Ap­ply by June 7

• Ryan Kidd 1 May 2026 23:24 UTC

  2 points

  0

  in reply to: Anna Thieser’s comment on: What holds AI safety to­gether? Co-au­thor­ship net­works from 200 papers

  Thanks for replying! I listed these organizations because they all maintain up-to-date repositories of papers they contributed to. If you added all papers linked there (and I think you should, as they are all AI safety papers), I suspect you would have ~400-500 papers, many of which would not be included in your initial 200!

  If I were running this project, I would additionally scrape papers from the websites of all the orgs listed on the AI safety map (they have a spreadsheet of orgs) and 80,000 Hours org list.

• Ryan Kidd 26 Apr 2026 22:39 UTC

  10 points

  4

  on: What holds AI safety to­gether? Co-au­thor­ship net­works from 200 papers

  Hi Anna, this looks like a great initiative! I think 200 papers seems quite small though. As an example, did you include all papers with contributions from MATS (~185 papers), EleutherAI, Apart, PIBBSS, Pivotal, SPAR, ERA, LASR, MARS, etc.? My impression is that a large amount of AI safety research is published in collaboration with nonprofit fellowship programs that borrow mentors from other institutions, which might help explain the apparent centrality of universities.

• Ryan Kidd 27 Mar 2026 23:46 UTC

  4 points

  2

  on: Where do AI Safety Fel­lows go? An­a­lyz­ing a dataset of 600+ alumni

  I found over 10% of fellows did another fellowship after their fellowship. This doesn’t feel enormously efficient.

  This matches my finding that 10% of each MATS cohort, on average, did MATS before. I don’t think this is particularly bad; it’s somewhat analogous to undergrads doing several internships in different fields or with different companies before graduating.

• Ryan Kidd 12 Mar 2026 22:52 UTC

  5 points

  1

  in reply to: Cleo Nardo’s comment on: The case for AI safety ca­pac­ity-build­ing work

  I now think the net-neutral ratio of new median safety to capabilities researchers is probably closer to 1:100 (though I think 1:10 isn’t crazy). However, the longitudinal MATS career data indicates our alumni are employed in safety to capabilities roles 8:1.

• Ryan Kidd 24 Feb 2026 23:20 UTC

  10 points

  3

  in reply to: Ryan Kidd’s comment on: Ryan Kidd’s Shortform

  My sense of priorities for an Australian AISI.

  Priorities

  1. Enable safe and secure frontier AI training and deployment in Australia

     1. Build pathways around copyright restriction that prevents frontier AI companies from building training datacenters in Australia; the OpenAI datacenter being built in Sydney is for inference only

     2. Frontier AI will change the nature of work, shape international power, and potentially change what it means to be human and Australia doesn’t have a seat at the table; fix this

     3. Create jobs and opportunities for Australians to build and contribute to safe and secure AI in Australia, rather than all going overseas

  2. Set clear national guidelines for AI safety, security, and transparency

     1. Set national AI standards similar to SB 53, RAISE Act, EU AI Act

     2. Create a “regulatory market” for AI and empower insurers, standard-setters, and auditors

     3. Allow beneficial AI development by removing regulatory uncertainty

  3. Advance Australian interests in AI safety and security internationally

     1. Contribute to frontier AI evaluations and safety research like UK AISI, US CAISI

     2. Leverage Australia’s role as a “middle power” to create beneficial AI outcomes

     3. Share resources with the international network of AI safety institutes

  Not priorities

  1. Overly focus on algorithmic bias, privacy violations, or misinformation/​deepfakes to the exclusion of systemic or catastrophic risk from AI

  2. Overly focus on non-frontier AI technologies like recommender systems or AI for drug discovery instead of frontier AI systems like ChatGPT, Claude, Gemini, Grok, Llama, DeepSeek, Kimi

  3. Focus on enforcing or extending copyright laws that prevent AI investment in Australia

• Ryan Kidd 23 Feb 2026 16:13 UTC

  8 points

  0

  in reply to: Ryan Kidd’s comment on: Ryan Kidd’s Shortform

  Credit to Daniel Eth, who inspired this.

• Ryan Kidd 22 Feb 2026 3:52 UTC

  4 points

  0

  in reply to: 142857’s comment on: Ryan Kidd’s Shortform

  Thanks, fixed!

• Ryan Kidd 22 Feb 2026 2:40 UTC

  115 points

  41

  on: Ryan Kidd’s Shortform

  Someone recently shared an interesting minimal model of electoral politics with me. Elected officials are influenced by two primary groups: voters and donors. Sometimes voter and donor interests align, but often they don’t. Voters are the ultimate deciders, but they are only really swayed by the top-3 most salient issues. On those issues, elected officials side with voters, but on the rest they tend to side with donors. It’s simply too costly for elected officials to go against donors on low-public salience issues; they can spend the donated money on campaigning for the top-salience issues, after all!

  One interesting low-salience issue is AI policy. Voters are generally anti-AI (they don’t like datacenters, automated surveillance, or recommender systems) but donors (often tech billionaires) are pro-AI. As AI is currently a low-salience issue for US voters (at least relative to the broader economy, healthcare, and immigration), politicians are siding with donors. For meaningful political action on AI, one would need either: AI to become a top-3 salience issue to voters (e.g., via job loss, weaponization, or warning shots); or more anti-AI donors to mobilize.

• Ryan Kidd 9 Feb 2026 20:14 UTC

  5 points

  0

  in reply to: Chris_Leong’s comment on: Ryan Kidd’s Shortform

  Yes, I would generally support picking the latter as they have a “faster time to mentorship/​research leadership/​impact” and the field seems currently bottlenecked on mentorship and research leads, not marginal engineers (though individual research leads might feel bottlenecked on marginal engineers).

  We should prioritize people who already have research or engineering experience or a very high iteration speed as we are operating under time constraints; AGI is coming soon. Additionally, I think “research taste” will be more important than engineering ability given AI automation and this takes a long time to build; better to select people with existing research experience they can adapt from another field (also promotes interdisciplinary knowledge transfer).

  I talk more about it here.

• Ryan Kidd 9 Feb 2026 2:52 UTC

  4 points

  2

  in reply to: BryceStansfield’s comment on: Ryan Kidd’s Shortform

  Yoshua Bengio, Paul Christiano, Geoffrey Irving seem more like technical AI safety experts than AI policy experts, but they arguably have strong influence on governments.

• Ryan Kidd 8 Feb 2026 23:27 UTC

  6 points

  3

  in reply to: Ryan Kidd’s comment on: Ryan Kidd’s Shortform

  What should be done? I think:

  • Office hubs: Expand SASS, which is in close proximity to the new OpenAI and Anthropic offices in Sydney. Start an AI safety hub in Canberra to support the new AISI. Successful AI safety hubs have benefited from prominent founding member organizations like ARC, CG, Redwood, and MIRI (for Constellation) and Apollo, BlueDot, and MATS (for LISA). Similarly, SASS should bring together orgs like the Gradient Institute and Harmony Intelligence in a shared space, and the new Canberra hub should be built around Good Ancestors. Office hubs can benefit member orgs by providing cheaper returns to scale, hosting shared networking events, facilitating collaboration, and providing a pipeline of strong new recruits in the form of office guests and members.

  • Training programs: Expand TARA and the Sydney AI Safety Fellowship program, focusing on accelerating top talent and building local mentorship capacity for future programs. Don’t focus on maximizing impact on participants; this is less important than reducing the mentorship bottleneck, which is best served by boosting the most advanced participants.

  • Academic labs: Build relationships with AI/​CS academics at UniMelb, Monash, USyd, ANU, UTS, UNSW, UA, UQ, etc. Help launch AI safety courses like Roy Rinberg and Boaz Barak did at Harvard. Other course inspiration is provided by Stanford and CAIS. Start AI safety academic labs like UC Berkeley CHAI, MIT AAG, NYU ARG, Bau Lab, Stanford HAI, CMU FOCAL, etc.

  • Conferences: Run an annual AI safety conference like the Australian AI Safety Forum 2024, bringing together academia, industry, government, and nonprofit field-builders. EAGx is probably not enough, as many people from academia, industry, and government likely won’t attend.

• Ryan Kidd 8 Feb 2026 23:04 UTC

  2 points

  0

  in reply to: aog’s comment on: Ryan Kidd’s Shortform

  I suspect that some LWers would interpret this as a (bad) argument for countries to build datacenters so they can exercise political control over AGI. I don’t think this works.

• Ryan Kidd 8 Feb 2026 20:08 UTC

  6 points

  4

  in reply to: aog’s comment on: Ryan Kidd’s Shortform

  Exactly! Also:

  • AI safety researchers tend to have reasonable views on AI safety and can serve as local advisors to governments, which probably trust foreign experts less.

  • Securing datacenters for SL5 to prevent proliferation of AGI is really hard and likely requires significant AI security expertise in government and local defence contractors.

  • A regulatory market approach to AI safety (possibly only useful pre-superintelligence) requires competent local auditors, standard-setters, and insurers.

• Ryan Kidd 7 Feb 2026 23:58 UTC

  80 points

  43

  on: Ryan Kidd’s Shortform

  AI safety field-building in Australia should accelerate. My rationale:

  • OpenAI opened a Sydney office in Dec 2025 and Anthropic is planning to open a Sydney office in 2026. These offices may hire safety staff from local talent, or partner with local auditing, evaluation, and security companies, including Harmony Intelligence, Good Ancestors, and Gradient Institute.

  • An Australian AISI was announced for early 2026 and is currently hiring. The UK AISI has benefited from close partnerships with Apollo Research, METR, and the LISA office community. There is a community space in Sydney, the Sydney AI Safety Space, and two field-building organizations, AI Safety ANZ and TARA, but these could expand substantially.

  • Australia seems like a prime location for datacenter build-out. OpenAI published an “AI blueprint” for Australia, calling for datacenter build-out, and started building a $4.6B datacenter in Sydney in Dec 2025. Australia is a NATO partner, Five Eyes member, and member of the AUKUS security partnership with the US and UK; it’s much more secure and aligned with US/​UK interests than Saudi Arabia. Australia is the second-largest exporter of thermal coal, has vast solar and wind resources, and the highest uranium reserves on earth. Australia is currently quite anti-nuclear at the moment, but it has no earthquakes or tsunamis to disrupt power plants. Janet Egan (CNAS) recently called for the development of US military AI projects in Australia, similar to the Pine Gap facility in the Northern Territory. AI safety & security research and political pressure for safety standards should focus on countries with frontier AI companies and datacenters.

  • Several prominent AI safety researchers have come from Australia, including Marcus Hutter, Buck Shlegeris, Jan Leike (PhD only), Ramana Kumar, Dan Murfet, and Daniel Filan. A decent number of MATS fellows come from Australia. Australian citizens can easily emigrate to the US (E-3 visa) or UK (YMS visa) for work. Seven of the top-100 computer science universities are in Australia.

  • Australia has a history as a middle power between the US and China, two of its largest trading partners. The country may play a significant role in international diplomacy efforts, such as a superintelligence ban or international AGI project.

  • Australia has a strong history of environmental activism, which might (or might not) be a useful asset for AI safety political mobilization.

• Ryan Kidd 27 Jan 2026 22:17 UTC

  3 points

  −2

  in reply to: Ryan Kidd’s comment on: Ryan Kidd’s Shortform

  I think that the distribution of mentors we are drawing from is slowing growing to include more highly respected academics and industry professionals by percentage. I think this increases the average quality of our mentor applicant pool, but I understand that this is might be controversial. Note that I still think our most impactful mentors are well-known within the AI safety field and most of the top-50 most impactful researchers in AI safety apply to mentor at MATS.

• Ryan Kidd 27 Jan 2026 20:50 UTC

  5 points

  0

  in reply to: Neel Nanda’s comment on: Ryan Kidd’s Shortform

  This is a hard question to answer precisely, as we have changed the metrics by which we have evaluated potential mentors several times. The average quality of mentors we accept has grown each program, by my lights. I weakly think that the average quality of mentors applying has also grown, though much slower.

• Ryan Kidd 27 Jan 2026 18:09 UTC

  12 points

  0

  on: Ryan Kidd’s Shortform

  Update on MATS applications trends:

  • Mentor applications seem to be growing 2.2x/​year and we accepted 20% as primary mentors for 10.0 (Summer 2026).

  • Fellow applications seem to be growing 1.8x/​year (counting only applicants who applied to at least one mentor) and we plan to accept 5% for 10.0.