This is a bit of a nit, but I don’t think METR has consumed that much of the “community resources”, especially of more experienced technical talent—I think only around two or three of current employees at METR were working infulltime roles on technical AI safety before they joined METR. This is a thing I care about and track in hiring—I don’t want to pull people away from doing other good work. (Edited to add the italics, prev claim was overstatement) (Second edit: although I still agree with the claim that we haven’t had a large negative impact on talent availability in technical AIS)
Don’t think I agree. When I scan down the staff, I recognize about half the names as having been around the AI safety scene for 4-8 years, either working on projects or seeking projects. You, Painter, Cotra, Filan, Wijk, Becker, Chan, Kinniment, Jurkovic, Von Arx, Kwa, Dhaliwal, Harris, Chen, have all been part of the AI safety community a long time, and would likely be working on another related project if not for this. Perhaps more that I’m not as immediately familiar with.
Added: To cache that out for those who don’t know who these people are:
Beth was an alignment researcher at OpenAI in 2019, and I know cared about this much earlier than that.
Chris Painter’s LinkedIn shows he worked on “AI Safety via debate” at OpenAI in 2019.
Ajeya Cotra has been working in the AI part of OpenPhil/Cog since at least 2018.
Daniel Filan was part of Stuart Russell’s CHAI at UCC Berkeley while he did his PhD there starting in 2016, and has hosted the AI X-Risk podcast since 2020.
Hjalmar Wijk was a MIRI Summer Fellow in 2019, also an FHI Summer Fellow that year, and I suspect was involved with FHI throughout his Oxford CS PhD in that time.
Joel Becker was AI Safety grantmaker with Manifund for 2 years before joining METR.
Lawrence Chan was also part of CHAI under Russell while doing his PhD there since 2018, then worked at Redwood and Alignment Research Center before METR.
Megan Kinniment was a summer research fellow at the FHI in 2020, did an AI project at the Center on Long-Term Risk that year, then continued to work between those two places before joining ARC.
Nikola Jurkovic was a research assistant on AI Safety projects at Harvard while a student there in 2023, did a bunch of work on the AI Safety Student Team, before joining METR.
Sydney Von Arx doesn’t have much online presence, but I know that Beth knows that Von Arx has been working on world-saving projects for a long-time (e.g. she cofounded the Open-Phil-funded Atlas Fellowship) and definitely has oriented to Superintelligent AI as the most important thing in the world since her time as part of Stanford EA.
Jasmine Dhaliwal was Open Philanthropy Chief of Staff for a year in 2023, then worked on FutureHouse, “A philanthropically-funded moonshot focused on building an AI Scientist”, before joining METR.
Kit Harris has been around the EA scene as long as I have, so at least a decade. He spent 7 years at Longview Philanthropy where amongst other things he “led grant investigations in artificial intelligence and biosecurity and laid the groundwork for new lines of work at Longview Philanthropy”.
Michael Chen was a research intern at Stuart Russell’s CHAI in 2022, and his METR profile says “Prior to joining METR, he contributed to research studying AI deception and hazardous knowledge in large language models.”
These are just the ones that I immediately recognized, I expect if I went through them all I’d find others have also been substantially involved (both professionally and personally) in the AI Safety scene prior to METR. And I count more than two or three people involved in technical AI safety in the above list.
I was thinking “was working FT on technical AIS before we hired them” more than “was around this space and might have done other AI safety things”—sorry if that was misleading. 1. You can count me although I also think I’m not central example of technical AIS work 2. Chris was mostly working on Alvea and policy stuff before METR, the debate thing was part-time contracting with me and not central example of technical AIS work 3. Ajeya—wasn’t necessarily counting grantmaking but that’s reasonable (also only joined METR very recently) 4. Daniel—was counting but I think not central example of FT TAIS work (also only joined METR very recently) 5. Hjalmar—hired partway through theoretical CS PhD, never had an FT AIS position I don’t think 6. Joel—pretty sure manifund grantmaking was not close to a FT position? 7. Lawrence—was counting 8. Megan—never had an FT AIS position I don’t think 9. Nikola—hired out of undergrad 10. Sydney—wasn’t counting as technical AIS 11. TKwa—not sure, was this FT position? 12. Jas—wasn’t counting as technical. Also I don’t think Future House counts as safety. 13. Kit—wasn’t counting as technical (he has math degree but I think fair to say the longview work is not central TAIS) 14. Michael—never had FT AIS position I don’t think David Rein who you missed I think is actually the clearest example
More than one of these people were at least temporarily unusually low-opp-cost for personal reasons that I don’t want to go into here (similar in spirit to ‘health/location constraints made it hard for them to have other jobs’)
In my mind there’s a big contrast here vs e.g. Ant, which I think has a huge number of people with multiple years experience working on technical AIS. E.g., people who I know off top of my head: Jon Uesato, Jeff Wu, Jan Leike, Chris Olah, Daniel Ziegler, Sam McCandlish, Jared Kaplan, Catherine Olsson, Amanda Askell, Tom Henighan, Shan Carter, Jan Kirchner, Nat McAleese, Carroll Wainright, Todor Markov, Dan Mossing, Steven Bills, William Saunders, Danny Hernandez, Dave Orr, Steven McAleer (all multiple years experience at OAI and/or GDM working on safety teams) Evan Hubinger, Sam Bowman, Sam Marks, Fabien Roger, Ethan Perez, Collin Burns, Akbir Khan, Tao Lin, Kshitij Sachan (previously working FT on safety in academia or nonprofits) (I expect I’m wrong about ~2 people in those lists)
There are probably a similar number more I’m uncertain about or are non-central examples like the METR ones discussed above.
I agree with your assessment here, I don’t think METR has had a significant negative effect on the availability of talent in the technical AGI Safety ecosystem, and Anthropic has had a massive negative one. GDM Safety has probably had a moderate negative one, offset by many people preferring to live in London
I see. Yes I think your previous claim was an overstatement.
I also share Habryka’s perspective, I’ve broadly not been sold on technical talent being vastly more important than non-technical talent since MIRI gave up on trying to actually solve the full alignment problem and Christiano stopped working on alignment theory, and I think that many of the people I listed have much more potential to do things that are good than most of the people you listed at Anthropic; but going into more detail on all that would take more time than seems worth it this afternoon.
FWIW I definitely don’t think technical talent is vastly more important, I just assumed that’s the resource that people would most think METR might be a large consumer of given most of our roles are technical roles
I was thinking “was working FT on technical AIS before we hired them” more than “was around this space and might have done other AI safety things”—sorry if that was misleading.
I think technical AI Safety work is among the less valuable kinds of work to do on the margin, so I definitely didn’t intend to constrain talent claims to technical AI safety. Indeed, generalist/entrepreneurial/communications talent seems a lot more valuable to me on the margin.
That said I agree that METR did not consume as much talent as Anthropic or OpenAI, and indeed many people went to work there to work on RSPs and similar if-then-commitment stuff, which didn’t pan out (and now my guess is they are very unlikely to leave). But METR + Apollo seem like the runner-ups right after the labs in terms of where people went to work (and at least at the time largely for if-then-commitment-like reasons).
This is a bit of a nit, but I don’t think METR has consumed that much of the “community resources”, especially of more experienced technical talent—I think only around two or three of current employees at METR were working in fulltime roles on technical AI safety before they joined METR. This is a thing I care about and track in hiring—I don’t want to pull people away from doing other good work.
(Edited to add the italics, prev claim was overstatement)
(Second edit: although I still agree with the claim that we haven’t had a large negative impact on talent availability in technical AIS)
Don’t think I agree. When I scan down the staff, I recognize about half the names as having been around the AI safety scene for 4-8 years, either working on projects or seeking projects. You, Painter, Cotra, Filan, Wijk, Becker, Chan, Kinniment, Jurkovic, Von Arx, Kwa, Dhaliwal, Harris, Chen, have all been part of the AI safety community a long time, and would likely be working on another related project if not for this. Perhaps more that I’m not as immediately familiar with.
Added: To cache that out for those who don’t know who these people are:
Beth was an alignment researcher at OpenAI in 2019, and I know cared about this much earlier than that.
Chris Painter’s LinkedIn shows he worked on “AI Safety via debate” at OpenAI in 2019.
Ajeya Cotra has been working in the AI part of OpenPhil/Cog since at least 2018.
Daniel Filan was part of Stuart Russell’s CHAI at UCC Berkeley while he did his PhD there starting in 2016, and has hosted the AI X-Risk podcast since 2020.
Hjalmar Wijk was a MIRI Summer Fellow in 2019, also an FHI Summer Fellow that year, and I suspect was involved with FHI throughout his Oxford CS PhD in that time.
Joel Becker was AI Safety grantmaker with Manifund for 2 years before joining METR.
Lawrence Chan was also part of CHAI under Russell while doing his PhD there since 2018, then worked at Redwood and Alignment Research Center before METR.
Megan Kinniment was a summer research fellow at the FHI in 2020, did an AI project at the Center on Long-Term Risk that year, then continued to work between those two places before joining ARC.
Nikola Jurkovic was a research assistant on AI Safety projects at Harvard while a student there in 2023, did a bunch of work on the AI Safety Student Team, before joining METR.
Sydney Von Arx doesn’t have much online presence, but I know that Beth knows that Von Arx has been working on world-saving projects for a long-time (e.g. she cofounded the Open-Phil-funded Atlas Fellowship) and definitely has oriented to Superintelligent AI as the most important thing in the world since her time as part of Stanford EA.
Thomas Kwa was a MIRI researcher in 2022.
Jasmine Dhaliwal was Open Philanthropy Chief of Staff for a year in 2023, then worked on FutureHouse, “A philanthropically-funded moonshot focused on building an AI Scientist”, before joining METR.
Kit Harris has been around the EA scene as long as I have, so at least a decade. He spent 7 years at Longview Philanthropy where amongst other things he “led grant investigations in artificial intelligence and biosecurity and laid the groundwork for new lines of work at Longview Philanthropy”.
Michael Chen was a research intern at Stuart Russell’s CHAI in 2022, and his METR profile says “Prior to joining METR, he contributed to research studying AI deception and hazardous knowledge in large language models.”
These are just the ones that I immediately recognized, I expect if I went through them all I’d find others have also been substantially involved (both professionally and personally) in the AI Safety scene prior to METR. And I count more than two or three people involved in technical AI safety in the above list.
I was thinking “was working FT on technical AIS before we hired them” more than “was around this space and might have done other AI safety things”—sorry if that was misleading.
1. You can count me although I also think I’m not central example of technical AIS work
2. Chris was mostly working on Alvea and policy stuff before METR, the debate thing was part-time contracting with me and not central example of technical AIS work
3. Ajeya—wasn’t necessarily counting grantmaking but that’s reasonable (also only joined METR very recently)
4. Daniel—was counting but I think not central example of FT TAIS work (also only joined METR very recently)
5. Hjalmar—hired partway through theoretical CS PhD, never had an FT AIS position I don’t think
6. Joel—pretty sure manifund grantmaking was not close to a FT position?
7. Lawrence—was counting
8. Megan—never had an FT AIS position I don’t think
9. Nikola—hired out of undergrad
10. Sydney—wasn’t counting as technical AIS
11. TKwa—not sure, was this FT position?
12. Jas—wasn’t counting as technical. Also I don’t think Future House counts as safety.
13. Kit—wasn’t counting as technical (he has math degree but I think fair to say the longview work is not central TAIS)
14. Michael—never had FT AIS position I don’t think
David Rein who you missed I think is actually the clearest example
More than one of these people were at least temporarily unusually low-opp-cost for personal reasons that I don’t want to go into here (similar in spirit to ‘health/location constraints made it hard for them to have other jobs’)
In my mind there’s a big contrast here vs e.g. Ant, which I think has a huge number of people with multiple years experience working on technical AIS.
E.g., people who I know off top of my head:
Jon Uesato, Jeff Wu, Jan Leike, Chris Olah, Daniel Ziegler, Sam McCandlish, Jared Kaplan, Catherine Olsson, Amanda Askell, Tom Henighan, Shan Carter, Jan Kirchner, Nat McAleese, Carroll Wainright, Todor Markov, Dan Mossing, Steven Bills, William Saunders, Danny Hernandez, Dave Orr, Steven McAleer (all multiple years experience at OAI and/or GDM working on safety teams)
Evan Hubinger, Sam Bowman, Sam Marks, Fabien Roger, Ethan Perez, Collin Burns, Akbir Khan, Tao Lin, Kshitij Sachan (previously working FT on safety in academia or nonprofits)
(I expect I’m wrong about ~2 people in those lists)
There are probably a similar number more I’m uncertain about or are non-central examples like the METR ones discussed above.
I agree with your assessment here, I don’t think METR has had a significant negative effect on the availability of talent in the technical AGI Safety ecosystem, and Anthropic has had a massive negative one. GDM Safety has probably had a moderate negative one, offset by many people preferring to live in London
I see. Yes I think your previous claim was an overstatement.
I also share Habryka’s perspective, I’ve broadly not been sold on technical talent being vastly more important than non-technical talent since MIRI gave up on trying to actually solve the full alignment problem and Christiano stopped working on alignment theory, and I think that many of the people I listed have much more potential to do things that are good than most of the people you listed at Anthropic; but going into more detail on all that would take more time than seems worth it this afternoon.
FWIW I definitely don’t think technical talent is vastly more important, I just assumed that’s the resource that people would most think METR might be a large consumer of given most of our roles are technical roles
I think technical AI Safety work is among the less valuable kinds of work to do on the margin, so I definitely didn’t intend to constrain talent claims to technical AI safety. Indeed, generalist/entrepreneurial/communications talent seems a lot more valuable to me on the margin.
That said I agree that METR did not consume as much talent as Anthropic or OpenAI, and indeed many people went to work there to work on RSPs and similar if-then-commitment stuff, which didn’t pan out (and now my guess is they are very unlikely to leave). But METR + Apollo seem like the runner-ups right after the labs in terms of where people went to work (and at least at the time largely for if-then-commitment-like reasons).
Hm I think UKAISI at least is a lot larger than METR or Apollo?
If you’re focusing more on generalist/entrepreneurial/communications skillsets then e.g. CG has more of these people than METR, I think?