Thanks for writing this! I’m curious if you have any information about the following questions:
What does the MATS team think are the most valuable research outputs from the program?
Which scholars was the MATS team most excited about in terms of their future plans/work?
IMO, these are the two main ways I would expect MATS to have impact: research output during the program and future research output/career trajectories of scholars.
Furthermore, I’d suspect things to be fairly tails-based (where EG the top 1-3 research outputs and the top 1-5 scholars are responsible for most of the impact).
Perhaps MATS as a program feels weird about ranking output or scholars so explicitly, or feels like it’s not their place.
But I think this kind of information seems extremely valuable. If I were considering whether or not I wanted to donate, for instance, my main questions would be “is the research good?” and “is the career development producing impactful people?” (as opposed to things like “what is the average rating on the EOY survey?”, though of course that information may matter for other purposes).
I also find it plausible that the top 1-5 scholars are responsible for most of the impact, and we want to investigate this to a greater extent. Unfortunately, it’s difficult to evaluate the impact of a scholar’s research and career trajectory until more like 3-12 months after the program, so we decided to separate that analysis from the retrospective of the summer 2023 program.
We’ve begun collecting this type of information (for past cohorts) via alumni surveys and other sources and hope to have another report out in the next few months that closer tracks the impact that we expect MATS to have.
these are the two main ways I would expect MATS to have impact: research output during the program and future research output/career trajectories of scholars.
We expect to achieve more impact through (2) than (1); per the theory of change section above, MATS’ mission is to expand the talent pipeline. Of course, we hope that scholars produce great research through the program, but often we are excited about scholars doing research (1) as a means for scholars tobecome better researchers (2). Other times, these goals are in tension. For example, some mentors explicitly de-emphasize research outputs, encouraging scholars to focus instead on pivoting frequently to develop better taste.
If you have your own reason to think (1) is comparably important to (2), I wonder if you also think providing mentees to researchers whose agendas you support is a similarly important source of MATS’ impact? From the Theory of Change section:
By mentoring MATS scholars, these senior researchers also benefit from research assistance
I would also suspect that #2 (finding/generating good researchers) is more valuable than #1 (generating or accelerating good research during the MATS program itself).
One problem with #2 is that it’s usually harder to evaluate and takes longer to evaluate. #2 requires projections, often over the course of years. #1 is still difficult to evaluate (what is “good alignment research” anyways?) but seems easier in comparison.
Also, I would expect correlations between #1 and #2. Like, one way to evaluate “how good are we doing at training researchers//who are the best researchers” is to ask “how good is the research they are producing//who produced the best research in this 3-month period?”
This process is (of course) imperfect. For example, someone might have great output because their mentor handed them a bunch of ready-to-go-projects, but the scholar didn’t actually have to learn the important skills of “forming novel ideas” or “figuring out how to prioritize between many different directions.”
But in general, I think it’s a pretty decent way to evaluate things. If someone has produced high-quality and original research during the MATS program, that sure does seem like a strong signal for their future potential. Likewise, in the opposite extreme, if during the entire summer cohort there were 0 instances of useful original work, that doesn’t necessarily mean something is wrong, but it would make me go “hmmm, maybe we should brainstorm possible changes to the program that could make it more likely that we see high-quality original output next time, and then we see how much those proposed changes trade-off against other desireada.”
(It seems quite likely to me that the MATS team has already considered all of this; just responding on the off-chance that something here is useful!)
Thanks for writing this! I’m curious if you have any information about the following questions:
What does the MATS team think are the most valuable research outputs from the program?
Which scholars was the MATS team most excited about in terms of their future plans/work?
IMO, these are the two main ways I would expect MATS to have impact: research output during the program and future research output/career trajectories of scholars.
Furthermore, I’d suspect things to be fairly tails-based (where EG the top 1-3 research outputs and the top 1-5 scholars are responsible for most of the impact).
Perhaps MATS as a program feels weird about ranking output or scholars so explicitly, or feels like it’s not their place.
But I think this kind of information seems extremely valuable. If I were considering whether or not I wanted to donate, for instance, my main questions would be “is the research good?” and “is the career development producing impactful people?” (as opposed to things like “what is the average rating on the EOY survey?”, though of course that information may matter for other purposes).
I also find it plausible that the top 1-5 scholars are responsible for most of the impact, and we want to investigate this to a greater extent. Unfortunately, it’s difficult to evaluate the impact of a scholar’s research and career trajectory until more like 3-12 months after the program, so we decided to separate that analysis from the retrospective of the summer 2023 program.
We’ve begun collecting this type of information (for past cohorts) via alumni surveys and other sources and hope to have another report out in the next few months that closer tracks the impact that we expect MATS to have.
We expect to achieve more impact through (2) than (1); per the theory of change section above, MATS’ mission is to expand the talent pipeline. Of course, we hope that scholars produce great research through the program, but often we are excited about scholars doing research (1) as a means for scholars to become better researchers (2). Other times, these goals are in tension. For example, some mentors explicitly de-emphasize research outputs, encouraging scholars to focus instead on pivoting frequently to develop better taste.
If you have your own reason to think (1) is comparably important to (2), I wonder if you also think providing mentees to researchers whose agendas you support is a similarly important source of MATS’ impact? From the Theory of Change section:
I would also suspect that #2 (finding/generating good researchers) is more valuable than #1 (generating or accelerating good research during the MATS program itself).
One problem with #2 is that it’s usually harder to evaluate and takes longer to evaluate. #2 requires projections, often over the course of years. #1 is still difficult to evaluate (what is “good alignment research” anyways?) but seems easier in comparison.
Also, I would expect correlations between #1 and #2. Like, one way to evaluate “how good are we doing at training researchers//who are the best researchers” is to ask “how good is the research they are producing//who produced the best research in this 3-month period?”
This process is (of course) imperfect. For example, someone might have great output because their mentor handed them a bunch of ready-to-go-projects, but the scholar didn’t actually have to learn the important skills of “forming novel ideas” or “figuring out how to prioritize between many different directions.”
But in general, I think it’s a pretty decent way to evaluate things. If someone has produced high-quality and original research during the MATS program, that sure does seem like a strong signal for their future potential. Likewise, in the opposite extreme, if during the entire summer cohort there were 0 instances of useful original work, that doesn’t necessarily mean something is wrong, but it would make me go “hmmm, maybe we should brainstorm possible changes to the program that could make it more likely that we see high-quality original output next time, and then we see how much those proposed changes trade-off against other desireada.”
(It seems quite likely to me that the MATS team has already considered all of this; just responding on the off-chance that something here is useful!)