I consider the central examples of successful AI safety org founders to be Redwood, METR, Transluce, GovAI, Apollo, FAR AI, MIRI, LawZero, Pattern Labs, CAIS, Goodfire, Palisade, BlueDot, Constellation, MATS, Horizon, etc. Broader-focus orgs like 80,000 Hours, Lightcone, CEA and others have also had large impact. Apologies to all those I’ve missed!
I definitely think founders should workshop their ideas a lot, but this is not necessarily the same thing as publishing original research or writing on forums. Caveat: research org founders often should be leading research papers.
I don’t think that a great founder will have more impact in scaling the AI safety research field by working at “Anthropic, GDM, or FAR Labs” relative to founding a new research org or training program.
Maybe I’m naive about how easy it is to adjust standards for grantmakers or training programs. My experience with MATS, LISA, and Manifund has involved a lot of selection and the bar at MATS has raised every program for 4 years now, but I don’t feel a lot of pressure from rejected applicants to lower our standards. Maybe this will come with time? Or maybe it’s an ecosystem-wide effect? I see the pressure to increase elite university admissions pressure as unideal, but not a field-killer; plus, AI safety seems farfrom this point. I acknowledge that you have a lot of experience with LTFF and other selection processes.
I don’t think AI companies scaling 2-3x/year is good for the world. I do think AI safety talent failing to keep up is bad for the world. It’s not so much an adversarial dynamic as a race to lower the alignment tax as much as possible at every stage.
I don’t think that Anthropic’s safety work is zero value. I’d like to see more people working on ASL-4,5 safety at Anthropic and Kimi, all else equal. I’d also like to see more AI safety training programs supplying talent, nonprofits orgs scaling auditing and research, and advocacy orgs shifting public perception.
I’m not sure how to think about CEA (and I lack your information here), but my first reaction is not “CEA should have been led by researchers.” I also don’t think Open Phil is a good example of an org that lacked researchers; some of the best worldview investigations research imo came from Open Phil staff or affiliates, including Joe Carlsmith, Ajeya Cotra, Holden Karnofsky, Carl Schulman, etc. (edit: which clearly informed OP grant making).
I’m more optimistic than you about the impact of encouraging more AI safety founders. I’m particularly excited by Halcyon Future’s work in helping launch Goodfire, AIUC, Lucid Computing, Transluce, Seismic, AVERI, Fathom, etc. To date, I know of only two such RL dataset startups that spawned via AI safety (Mechanize, Calaveras) in contrast to ~150 AI safety-promoting orgs (though I’m sure there are other examples of AI safety-detracting startups).
I fully endorse more potential founders writing up pitches or theories of change for discussion on LW or founder networks! I think this can only strengthen their impact.
the bar at MATS has raised every program for 4 years now
What?! Something terrible must be going on in your mechanisms for evaluating people (which to be clear, isn’t surprising, indeed, you are the central target of the optimization that is happening here, but like, to me it illustrates the risks here quite cleanly).
It is very very obvious to me that median MATS participant quality has gone down continuously for the last few cohorts. I thought this was somewhat clear to y’all and you thought it was worth the tradeoff of having bigger cohorts, but you thinking it has “gone up continuously” shows a huge disconnect.
Like, these days at the end of a MATS program half of the people couldn’t really tell you why AI might be an existential risk at all. Their eyes glaze over when you try to talk about AI strategy. IDK, maybe these people are better ML researchers, but obviously they are worse contributors to the field than the people in the early cohorts.
Yeah, I mean, I do think I am a lot more pessimistic about all of these. If you want we can make a bet on how well things have played out with these in 5 years, deferring to some small panel of trusted third party people.
To date, I know of only two such RL dataset startups that spawned via AI safety
Agree. Making RL environments/datasets has only very recently become a highly profitable thing, so you shouldn’t expect much! I am happy to make bets that we will see many more in the next 1-2 years.
The MATS acceptance rate was 33% in Summer 2022 (the first program with open applications) and decreased to 4.3% (in terms of first-stage applicants; ~7% if you only count those who completed all stages) in Summer 2025. Similarly, our mentor acceptance rate decreased from 100% in Summer 2022 to 27% for the upcoming Winter 2026 Program.
I don’t have plots prepared, but measures of scholar technical ability (e.g., mentor ratings, placements, CodeSignal score) have consistently increased. I feel very confident that MATS is consistently improving in our ability to find, train, and place ML (and other) researchers in AI safety roles, predominantly as “Iterators”. Also, while the fraction of the cohort that display strong “Connector” disposition seems to have decreased over time, I think that the raw number of strong Connectors has generally increased with program size due to our research diversity metric in mentor selection. I would argue that the phenomenon you are witnessing is an increasing pivot from more theoretical to empirical AI safety mentors and research agendas.
Based on my personal experience, I think the claim “half of MATS couldn’t tell you why AI might be an existential risk” is incorrect. I can’t speak to how MATS scholars have engaged with you on AI strategy, but I would bet that the average MATS scholar today spends a lot more time on ML experiments than reading AI safety strategy docs compared to three years ago. To be clear, I think this is a good thing! I respect your disagreement here. MATS has tried to run AI safety strategy workshops and readinggroups many times in the past, but this has generally had low engagement relative to our seminar series (which features some prominent AI safety strategists anyways). If you have great ideas for how to better structure strategy workshops or generate interest, I would love to hear! (We are currently brainstorming this.)
The MATS acceptance rate was 33% in Summer 2022 (the first program with open applications) and decreased to 4.3% (in terms of first-stage applicants; ~7% if you only count those who completed all stages) in Summer 2025. Similarly, our mentor acceptance rate decreased from 100% in Summer 2022 to 27% for the upcoming Winter 2026 Program.
I mean, in as much as one is worried about Goodhart’s law, and the issue in contention is adversarial selection, then the acceptance rate going down over time is kind of the premise of the conversation. Like, it would be evidence against my model of the situation if the acceptance rate had been going up (since that would imply MATS is facing less adversarial pressure over time).
I don’t have plots prepared, but measures of scholar technical ability (e.g., mentor ratings, placements, CodeSignal score) have consistently increased. I feel very confident that MATS is consistently improving in our ability to find, train, and place ML (and other) researchers in AI safety roles, predominantly as “Iterators”.
Mentor ratings is the most interesting category to me. As you can imagine I don’t care much for ML skill at the margin. CodeSignal is a bit interesting though I am not familiar enough with it to interpret it, but I might look into it.
I don’t know whether you have any plots of mentor ratings over time broken out by individual mentor. My best guess is the reason why mentor ratings are going up is because you have more mentors who are looking for basically just ML skill, and you have successfully found a way to connect people into ML roles.
This is of course where most of your incentive gradient was pointing to in the first place, as of course the entities that are just trying to hire ML researchers have the most resources, and you will get the most applicants for highly paid industry ML roles, which are currently among the most prestigious and most highly paid roles in the world (while of course being centrally responsible for the risk from AI that we are working on).
In regards to adversarial selection, we can compare MATS to SPAR. SPAR accepted ~300 applicants in their latest batch, ~3x MATS (it’s easier to scale if you’re remote, don’t offer stipends, and allow part-timers). I would bet that the average research impact of SPAR participants is significantly lower than that of MATS, though there might be plenty of confounders here. It might be worth doing a longitudinal study here comparing various training programs’ outcomes over time, including PIBBSS, ERA, etc.
I think your read of the situation re. mentor ratings is basically correct: increasingly many MATS mentors primarily care about research execution ability (generally ML), not AI safety strategy knowledge. I see this as a feature, not a bug, but I understand why you disagree. I think you are prioritizing a different skillset than most mentors that our mentor selection committee rates highly. Interestingly, most of the technical mentors that you rate highly seem to primarily care about object-level research ability and think that strategy/research taste can be learned on the job!
Note that I think the pendulum might start to swing back towards mentors valuing high-level AI safety strategy knowledge as the Iterator archetype is increasingly replaced/supplemented by AI. The Amplifier archetype seems increasingly in-demand as orgs scale, and we might see a surge in Connectors as AI agents improve to the point that their theoretical ideas are more testable. Also note that we might have different opinions on the optimal ratio of “visionaries” vs. “experimenters” in an emerging research field.
I would bet that the average research impact of SPAR participants is significantly lower than that of MATS
I mean, sure? I am not saying your selection is worse than useless and it would be better for you to literally accept all of them, that would clearly also be bad for MATS.
I think you are prioritizing a different skillset than most mentors that our mentor selection committee rates highly. Interestingly, most of the technical mentors that you rate highly seem to primarily care about object-level research ability and think that strategy/research taste can be learned on the job!
I mean, there are obvious coordination problems here. In as much as someone is modeling MATS as a hiring pipeline, and not necessarily the one most likely to produce executive-level talent, you will have huge amounts of pressure to produce line-worker talent. This doesn’t mean the ecosystem doesn’t need executive-level talent (indeed, this post is partially about how we need more), but of course large scaling organizations create more pressure for line-working talent.
Two other issues with this paragraph:
Yes, I don’t think strategic judgement generally commutes. Most MATS mentors who I think are doing good research don’t necessarily themselves know what’s most important for the field.
I agree with the purported opinion that strategy/research taste can often be learned on the job. But I do feel very doomy about recruiting people who don’t seem to care deeply about x-risk. I would be kind of surprised if the mentors I am most excited about don’t have the same opinion, but it would be an interesting update if so!
Note that I think the pendulum might start to swing back towards mentors valuing high-level AI safety strategy knowledge as the Iterator archetype is increasingly replaced/supplemented by AI. The Amplifier archetype seems increasingly in-demand as orgs scale, and we might see a surge in Connectors as AI agents improve to the point that their theoretical ideas are more testable. Also note that we might have different opinions on the optimal ratio of “visionaries” vs. “experimenters” in an emerging research field.
I don’t particularly think these “archetypes” are real or track much of the important dimensions, so I am not really sure what you are saying here.
A few quick comments, on the same theme as but mostly unrelated to the exchange so far:
I’m not very sold on “cares about xrisk” as a key metric for technical researchers. I am more interested in people who want to very deeply understand how intelligence works (whether abstractly or in neural networks in particular). I think the former is sometimes a good proxy for the latter but it’s important not to conflate them. See this post for more.
Having said that, I don’t get much of a sense that many MATS scholars want to deeply understand how intelligence works. When I walked around the poster showcase at the most recent iteration of MATS, a large majority of the projects seemed like they’d prioritized pretty “shallow” investigations. Obviously it’s hard to complete deep scientific work in three months but at least on a quick skim I didn’t see many projects that seemed like they were even heading in that direction. (I’d cite Tom Ringstrom as one example of a MATS scholar who was trying to do deep and rigorous work, though I also think that his core assumptions are wrong.)
As one characterization of an alternative approach: my intership with Owain Evans back in 2017 consisted of me basically sitting around and thinking about AI safety for three months. I had some blog posts as output but nothing particularly legible. I think this helped nudge me towards thinking more deeply about AI safety subsequently (though it’s hard to assign specific credit).
There’s an incentive alignment problem where even if mentors want scholars to spend their time thinking carefully, the scholars’ careers will benefit most from legible projects. In my most recent MATS cohort I’ve selected for people who seem like they would be happy to just sit around and think for the whole time period without feeling much internal pressure to produce legible outputs. We’ll see how that goes.
Hmm, I was referring here to “who I would want to hire at Lightcone” (and similarly, who I expect other mentors would be interested in hiring for their orgs) where I do think I would want to hire people who are on board with that organizational mission.
At the field level, I think we probably still have some disagreement about how valuable people caring about the AI X-risk case is, but I feel a lot less strongly about it, and think I could end up pretty excited about a MATS-like program that is more oriented around doing ambitious understanding of the nature of intelligence.
As an atypical applicant to MATS (no PhD, no coding/ technical skills, not early career, new to AI), I found it incredibly difficult to find mentors who were looking to hold space for just thinking about intelligence. I’d have loved to apply to a stream that involved just thinking, writing, being challenged and repeating until I’d a thesis worth pursuing. To me, it seemed more like most mentors were looking to test very specific hypothesis, and maybe it’s for all the reasons you’ve stated above. But for someone new and inexperienced, I felt pretty unsure about applying at all.
The MATS acceptance rate was 33% in Summer 2022 (the first program with open applications) and decreased to 4.3% (in terms of first-stage applicants; ~7% if you only count those who completed all stages) in Summer 2025. Similarly, our mentor acceptance rate decreased from 100% in Summer 2022 to 27% for the upcoming Winter 2026 Program.
This is not counter-evidence to the accusation that scholar quality has been going downhill unless you add in several other assumptions.
“To be clear, I think this is a good thing! I respect your disagreement here. MATS has tried to run AI safety strategy workshops and reading groups many times in the past, but this has generally had low engagement relative to our seminar series”
I suspect that achieving high-engagement will be hard because fellows have to compete for extension funding.
I might have a special view here since I did MATS 4.0 and 8.0.
I think I met some excellent people at MATS 8.0 but would not say they are stronger than 4.0, my guess is that quality went down slightly. I remember in 4.0 a few people that impressed me quite a lot, which I saw less in 8.0. (4.0 had more very incompetent people though)
at the end of a MATS program half of the people couldn’t really tell you why AI might be an existential risk at all.
I think this is sadly somewhat true, I talked with some people in 8.0 who didn’t seem to have any particular concern with AI existential risk or seemingly never really thought about that. However, I think most people were in fact very concerned about AI existential risk. I ran a poll at some point about Eliezer’s new book and a significant minority of students seemed to have pre-ordered Eleizer’s book, which I guess is a pretty good proxy for whether someone is seriously engaging with AI X-risk.
My guess is that the recruitment process might need another variable to measure rather than academics/coding/ml experience. The kind of thing that Tim Hua (8.0 scholar) has who created an AI psychosis bench.
Also it seems to me that if you build an organization that tries to fight against the end of the world from AI, somebody should say that. Might put off some people and perhaps that should happens early. Maybe the website should say: “AI could kill literally everyone, let’s try to do something!”. And maybe the people who heard this MATS thing is good to have on their CV to apply to a PhD or a lab to land a high paying job eventually would be put off by that.
Perhaps there should also be a test where people don’t have internet access and have to answer some basic alignment questions: like why could a system that we optimize with RL develop power seeking drives? Why might training an AI create weird unpredictable preferences in an AI?
My guess at what’s happening here: for the first iterations of MATS (think MATS 2.0 at the Lightcone WeWork) you would have folks who were already into AI Safety for quite a long time and were interested in doing some form of internship-like thing for a summer. But as you run more cohorts (and make the cohorts bigger) then the density of people who have been interested in safety for a long time naturally decreases (because all the people who were interested in safety for years already applied to previous iterations).
Thanks for reading and replying! I’ll be brief:
I consider the central examples of successful AI safety org founders to be Redwood, METR, Transluce, GovAI, Apollo, FAR AI, MIRI, LawZero, Pattern Labs, CAIS, Goodfire, Palisade, BlueDot, Constellation, MATS, Horizon, etc. Broader-focus orgs like 80,000 Hours, Lightcone, CEA and others have also had large impact. Apologies to all those I’ve missed!
I definitely think founders should workshop their ideas a lot, but this is not necessarily the same thing as publishing original research or writing on forums. Caveat: research org founders often should be leading research papers.
I don’t think that a great founder will have more impact in scaling the AI safety research field by working at “Anthropic, GDM, or FAR Labs” relative to founding a new research org or training program.
Maybe I’m naive about how easy it is to adjust standards for grantmakers or training programs. My experience with MATS, LISA, and Manifund has involved a lot of selection and the bar at MATS has raised every program for 4 years now, but I don’t feel a lot of pressure from rejected applicants to lower our standards. Maybe this will come with time? Or maybe it’s an ecosystem-wide effect? I see the pressure to increase elite university admissions pressure as unideal, but not a field-killer; plus, AI safety seems far from this point. I acknowledge that you have a lot of experience with LTFF and other selection processes.
I don’t think AI companies scaling 2-3x/year is good for the world. I do think AI safety talent failing to keep up is bad for the world. It’s not so much an adversarial dynamic as a race to lower the alignment tax as much as possible at every stage.
I don’t think that Anthropic’s safety work is zero value. I’d like to see more people working on ASL-4,5 safety at Anthropic and Kimi, all else equal. I’d also like to see more AI safety training programs supplying talent, nonprofits orgs scaling auditing and research, and advocacy orgs shifting public perception.
I’m not sure how to think about CEA (and I lack your information here), but my first reaction is not “CEA should have been led by researchers.” I also don’t think Open Phil is a good example of an org that lacked researchers; some of the best worldview investigations research imo came from Open Phil staff or affiliates, including Joe Carlsmith, Ajeya Cotra, Holden Karnofsky, Carl Schulman, etc. (edit: which clearly informed OP grant making).
I’m more optimistic than you about the impact of encouraging more AI safety founders. I’m particularly excited by Halcyon Future’s work in helping launch Goodfire, AIUC, Lucid Computing, Transluce, Seismic, AVERI, Fathom, etc. To date, I know of only two such RL dataset startups that spawned via AI safety (Mechanize, Calaveras) in contrast to ~150 AI safety-promoting orgs (though I’m sure there are other examples of AI safety-detracting startups).
I fully endorse more potential founders writing up pitches or theories of change for discussion on LW or founder networks! I think this can only strengthen their impact.
What?! Something terrible must be going on in your mechanisms for evaluating people (which to be clear, isn’t surprising, indeed, you are the central target of the optimization that is happening here, but like, to me it illustrates the risks here quite cleanly).
It is very very obvious to me that median MATS participant quality has gone down continuously for the last few cohorts. I thought this was somewhat clear to y’all and you thought it was worth the tradeoff of having bigger cohorts, but you thinking it has “gone up continuously” shows a huge disconnect.
Like, these days at the end of a MATS program half of the people couldn’t really tell you why AI might be an existential risk at all. Their eyes glaze over when you try to talk about AI strategy. IDK, maybe these people are better ML researchers, but obviously they are worse contributors to the field than the people in the early cohorts.
Yeah, I mean, I do think I am a lot more pessimistic about all of these. If you want we can make a bet on how well things have played out with these in 5 years, deferring to some small panel of trusted third party people.
Agree. Making RL environments/datasets has only very recently become a highly profitable thing, so you shouldn’t expect much! I am happy to make bets that we will see many more in the next 1-2 years.
I feel actively excited about 2 of these, quite negative about 1 of them, and confused/neutral about the others.
Can you share which?
The MATS acceptance rate was 33% in Summer 2022 (the first program with open applications) and decreased to 4.3% (in terms of first-stage applicants; ~7% if you only count those who completed all stages) in Summer 2025. Similarly, our mentor acceptance rate decreased from 100% in Summer 2022 to 27% for the upcoming Winter 2026 Program.
I don’t have plots prepared, but measures of scholar technical ability (e.g., mentor ratings, placements, CodeSignal score) have consistently increased. I feel very confident that MATS is consistently improving in our ability to find, train, and place ML (and other) researchers in AI safety roles, predominantly as “Iterators”. Also, while the fraction of the cohort that display strong “Connector” disposition seems to have decreased over time, I think that the raw number of strong Connectors has generally increased with program size due to our research diversity metric in mentor selection. I would argue that the phenomenon you are witnessing is an increasing pivot from more theoretical to empirical AI safety mentors and research agendas.
Based on my personal experience, I think the claim “half of MATS couldn’t tell you why AI might be an existential risk” is incorrect. I can’t speak to how MATS scholars have engaged with you on AI strategy, but I would bet that the average MATS scholar today spends a lot more time on ML experiments than reading AI safety strategy docs compared to three years ago. To be clear, I think this is a good thing! I respect your disagreement here. MATS has tried to run AI safety strategy workshops and reading groups many times in the past, but this has generally had low engagement relative to our seminar series (which features some prominent AI safety strategists anyways). If you have great ideas for how to better structure strategy workshops or generate interest, I would love to hear! (We are currently brainstorming this.)
I mean, in as much as one is worried about Goodhart’s law, and the issue in contention is adversarial selection, then the acceptance rate going down over time is kind of the premise of the conversation. Like, it would be evidence against my model of the situation if the acceptance rate had been going up (since that would imply MATS is facing less adversarial pressure over time).
Mentor ratings is the most interesting category to me. As you can imagine I don’t care much for ML skill at the margin. CodeSignal is a bit interesting though I am not familiar enough with it to interpret it, but I might look into it.
I don’t know whether you have any plots of mentor ratings over time broken out by individual mentor. My best guess is the reason why mentor ratings are going up is because you have more mentors who are looking for basically just ML skill, and you have successfully found a way to connect people into ML roles.
This is of course where most of your incentive gradient was pointing to in the first place, as of course the entities that are just trying to hire ML researchers have the most resources, and you will get the most applicants for highly paid industry ML roles, which are currently among the most prestigious and most highly paid roles in the world (while of course being centrally responsible for the risk from AI that we are working on).
In regards to adversarial selection, we can compare MATS to SPAR. SPAR accepted ~300 applicants in their latest batch, ~3x MATS (it’s easier to scale if you’re remote, don’t offer stipends, and allow part-timers). I would bet that the average research impact of SPAR participants is significantly lower than that of MATS, though there might be plenty of confounders here. It might be worth doing a longitudinal study here comparing various training programs’ outcomes over time, including PIBBSS, ERA, etc.
I think your read of the situation re. mentor ratings is basically correct: increasingly many MATS mentors primarily care about research execution ability (generally ML), not AI safety strategy knowledge. I see this as a feature, not a bug, but I understand why you disagree. I think you are prioritizing a different skillset than most mentors that our mentor selection committee rates highly. Interestingly, most of the technical mentors that you rate highly seem to primarily care about object-level research ability and think that strategy/research taste can be learned on the job!
Note that I think the pendulum might start to swing back towards mentors valuing high-level AI safety strategy knowledge as the Iterator archetype is increasingly replaced/supplemented by AI. The Amplifier archetype seems increasingly in-demand as orgs scale, and we might see a surge in Connectors as AI agents improve to the point that their theoretical ideas are more testable. Also note that we might have different opinions on the optimal ratio of “visionaries” vs. “experimenters” in an emerging research field.
I mean, sure? I am not saying your selection is worse than useless and it would be better for you to literally accept all of them, that would clearly also be bad for MATS.
I mean, there are obvious coordination problems here. In as much as someone is modeling MATS as a hiring pipeline, and not necessarily the one most likely to produce executive-level talent, you will have huge amounts of pressure to produce line-worker talent. This doesn’t mean the ecosystem doesn’t need executive-level talent (indeed, this post is partially about how we need more), but of course large scaling organizations create more pressure for line-working talent.
Two other issues with this paragraph:
Yes, I don’t think strategic judgement generally commutes. Most MATS mentors who I think are doing good research don’t necessarily themselves know what’s most important for the field.
I agree with the purported opinion that strategy/research taste can often be learned on the job. But I do feel very doomy about recruiting people who don’t seem to care deeply about x-risk. I would be kind of surprised if the mentors I am most excited about don’t have the same opinion, but it would be an interesting update if so!
I don’t particularly think these “archetypes” are real or track much of the important dimensions, so I am not really sure what you are saying here.
A few quick comments, on the same theme as but mostly unrelated to the exchange so far:
I’m not very sold on “cares about xrisk” as a key metric for technical researchers. I am more interested in people who want to very deeply understand how intelligence works (whether abstractly or in neural networks in particular). I think the former is sometimes a good proxy for the latter but it’s important not to conflate them. See this post for more.
Having said that, I don’t get much of a sense that many MATS scholars want to deeply understand how intelligence works. When I walked around the poster showcase at the most recent iteration of MATS, a large majority of the projects seemed like they’d prioritized pretty “shallow” investigations. Obviously it’s hard to complete deep scientific work in three months but at least on a quick skim I didn’t see many projects that seemed like they were even heading in that direction. (I’d cite Tom Ringstrom as one example of a MATS scholar who was trying to do deep and rigorous work, though I also think that his core assumptions are wrong.)
As one characterization of an alternative approach: my intership with Owain Evans back in 2017 consisted of me basically sitting around and thinking about AI safety for three months. I had some blog posts as output but nothing particularly legible. I think this helped nudge me towards thinking more deeply about AI safety subsequently (though it’s hard to assign specific credit).
There’s an incentive alignment problem where even if mentors want scholars to spend their time thinking carefully, the scholars’ careers will benefit most from legible projects. In my most recent MATS cohort I’ve selected for people who seem like they would be happy to just sit around and think for the whole time period without feeling much internal pressure to produce legible outputs. We’ll see how that goes.
Hmm, I was referring here to “who I would want to hire at Lightcone” (and similarly, who I expect other mentors would be interested in hiring for their orgs) where I do think I would want to hire people who are on board with that organizational mission.
At the field level, I think we probably still have some disagreement about how valuable people caring about the AI X-risk case is, but I feel a lot less strongly about it, and think I could end up pretty excited about a MATS-like program that is more oriented around doing ambitious understanding of the nature of intelligence.
Sounds like PIBBSS/PrincInt!
As an atypical applicant to MATS (no PhD, no coding/ technical skills, not early career, new to AI), I found it incredibly difficult to find mentors who were looking to hold space for just thinking about intelligence. I’d have loved to apply to a stream that involved just thinking, writing, being challenged and repeating until I’d a thesis worth pursuing. To me, it seemed more like most mentors were looking to test very specific hypothesis, and maybe it’s for all the reasons you’ve stated above. But for someone new and inexperienced, I felt pretty unsure about applying at all.
This is not counter-evidence to the accusation that scholar quality has been going downhill unless you add in several other assumptions.
It’s not supposed to be counter-evidence in its own right. I like to present the full picture.
“To be clear, I think this is a good thing! I respect your disagreement here. MATS has tried to run AI safety strategy workshops and reading groups many times in the past, but this has generally had low engagement relative to our seminar series”
I suspect that achieving high-engagement will be hard because fellows have to compete for extension funding.
True, but we accepted 75% of all scholars into the 6-month extension last program, so the pressure might not be that large now.
What percentage applied?
I might have a special view here since I did MATS 4.0 and 8.0.
I think I met some excellent people at MATS 8.0 but would not say they are stronger than 4.0, my guess is that quality went down slightly. I remember in 4.0 a few people that impressed me quite a lot, which I saw less in 8.0. (4.0 had more very incompetent people though)
I think this is sadly somewhat true, I talked with some people in 8.0 who didn’t seem to have any particular concern with AI existential risk or seemingly never really thought about that. However, I think most people were in fact very concerned about AI existential risk. I ran a poll at some point about Eliezer’s new book and a significant minority of students seemed to have pre-ordered Eleizer’s book, which I guess is a pretty good proxy for whether someone is seriously engaging with AI X-risk.
My guess is that the recruitment process might need another variable to measure rather than academics/coding/ml experience. The kind of thing that Tim Hua (8.0 scholar) has who created an AI psychosis bench.
Also it seems to me that if you build an organization that tries to fight against the end of the world from AI, somebody should say that. Might put off some people and perhaps that should happens early. Maybe the website should say: “AI could kill literally everyone, let’s try to do something!”. And maybe the people who heard this MATS thing is good to have on their CV to apply to a PhD or a lab to land a high paying job eventually would be put off by that.
Perhaps there should also be a test where people don’t have internet access and have to answer some basic alignment questions: like why could a system that we optimize with RL develop power seeking drives? Why might training an AI create weird unpredictable preferences in an AI?
My guess at what’s happening here: for the first iterations of MATS (think MATS 2.0 at the Lightcone WeWork) you would have folks who were already into AI Safety for quite a long time and were interested in doing some form of internship-like thing for a summer. But as you run more cohorts (and make the cohorts bigger) then the density of people who have been interested in safety for a long time naturally decreases (because all the people who were interested in safety for years already applied to previous iterations).