I’ve talked to a lot of people who have left leading AI companies for reasons related to thinking that their company was being insufficiently cautious. I wouldn’t usually say that they’d left “in protest”; for example, most of them haven’t directly criticized the companies after leaving.
In my experience, the main reason that most of these people left was that they found it very unpleasant to working there and thought their research would be better elsewhere, not that they wanted to protest poor safety policies per se. I usually advise such people against leaving if the company has very few safety staff, but it depends on their skillset.
(These arguments don’t apply to Anthropic: there are many people there who I think will try to implement reasonable safety techniques, so on the current margin, the benefits to safety technique implementation of marginal people seems way lower. It might still make sense to work at Anthropic, especially if you think it’s a good place to do safety research that can be exported.)
Incidentally, I’m happy to talk to people who are considering leaving AI companies and give them much more specific advice.
My impression is that few (one or two?) of the safety people who have quit a leading lab did so to protest poor safety policies, and of those few none saw staying as a viable option.
Relatedly, I think Buck far overestimates the influence and resources of safety-concerned staff in a ‘rushed unreasonable developer’.
My impression is that few (one or two?) of the safety people who have quit a leading lab did so to protest poor safety policies, and of those few none saw staying as a viable option.
While this isn’t amazing evidence, my sense is there have been around 6 people who quit who in-parallel to them announcing their leave called out OpenAI’s reckless attitude towards risk (at various levels of explicitness, but quite strongly in all cases by standard professional norms).
It’s hard to say that people quit “to protest safety policies”, but they definitely used their leaving to protest safety policies. My sense is almost everyone who left in the last year (Daniel, William, Richard, Steven Adler, Miles) did so with a pretty big public message.
I don’t think Miles’ or Richard’s stated reasons for resigning included safety policies, for example.
But my broader point is that “fewer safety people should quit leading labs to protest poor safety policies” is basically a non-sequitor from “people have quit leading labs because they think they’ll be more effective elsewhere”, whether because they want to do something different or independent, or because they no longer trust the lab to behave responsibly.
Hmm, you have a very different read of Richard’s message than I do. I agree Miles’ statement did not reason through safety policies, but IMO his blogging since then has included a lot of harsh words for OpenAI, in a way that at least to me made the connection clear (and I think also to many others, but IDK, it’s still doing some tea-leaf reading).
FWIW I think of “OpenAI leadership being untrustworthy” (a significant factor in me leaving) as different from “OpenAI having bad safety policies” (not a significant factor in me leaving). Not sure if it matters, I expect that Scott was using “safety policies” more expansively than I do. But just for the sake of clarity:
I am generally pretty sympathetic to the idea that it’s really hard to know what safety policies to put in place right now. Many policies pushed by safety people (including me, in the past) have been mostly kayfabe (e.g. being valuable as costly signals, not on the object level). There are a few object-level safety policies that I really wish OpenAI would do right now (most clearly, implementing better security measures) but I didn’t leave because of that (if I had, I would have tried harder to check before I left what security measures OpenAI did have, made specific objections internally about them before I left, etc).
This may just be a semantic disagreement, it seems very reasonable to define “don’t make employees sign non-disparagements” as a safety policy. But in my mind at least stuff like that is more of a lab governance policy (or maybe a meta-level safety policy).
Relatedly, I think Buck far overestimates the influence and resources of safety-concerned staff in a ‘rushed unreasonable developer’.
As in, you don’t expect they’ll be able to implement stuff even if it doesn’t make anyone’s workflow harder or you don’t expect they’ll be able to get that much compute?
Naively, we might expect ~1% of compute as we might expect around 1000 researchers and 10/1000 is 1%. Buck said 3% because I argued for increasing this number. My case would be that there will be bunch of cases where the thing they want to do is obviously reasonable and potentially justifiable from multiple perspectives (do some monitoring of internal usage, fine-tune a model for forecasting/advice, use models to do safety research) such that they can pull somewhat more compute than just the head count would suggest.
I also agree with Zac, maybe if you had a really well-selected group of 10 people you could do something, but 10 randomly selected AGI safety researchers probably don’t accomplish much.
By far my biggest objection is that there are approximately zero useful things that “[don’t] make anyone’s workflow harder”. I expect you’re vastly underestimating the complexity of production systems and companies that build them, and the number of constraints they are under. (You are assuming a do-ocracy though, depending on how much of a do-ocracy it is (e.g. willing to ignore laws) I could imagine changing my mind here.)
EDIT: I could imagine doing asynchronous monitoring of internal deployments. This is still going to make some workflows harder, but probably not a ton, so it seems surmountable. Especially since you could combine it with async analyses that the unreasonable developer actually finds useful.
EDIT 2 (Feb 7): To be clear, I also disagree with the compute number. I’m on board with starting with 1% since they are 1% of the headcount. But then I would decrease it first because they’re not a high-compute capabilities team, and second because whatever they are doing should be less useful to the company than whatever the other researchers are doing (otherwise why weren’t the other researchers doing it?), so maybe I’d estimate 0.3%. But this isn’t super cruxy because I think you can do useful safety work with just 0.3% of the compute.
Again, I could imagine getting more compute with a well-selected group of 10 people (though even then 3% seems unlikely, I’m imagining more like 1%), but I don’t see why in this scenario you should assume you get a well-selected group, as opposed to 10 random AGI safety researchers.
Yep, I think that at least some of the 10 would have to have some serious hustle and political savvy that is atypical (but not totally absent) among AI safety people.
What laws are you imagine making it harder to deploy stuff? Notably I’m imagining these people mostly doing stuff with internal deployments.
I think you’re overfixating on the experience of Google, which has more complicated production systems than most.
I agree with Rohin that there are approximately zero useful things that don’t make anyone’s workflow harder. The default state is “only just working means working, so I’ve moved on to the next thing” and if you want to change something there’d better be a benefit to balance the risk of breaking it.
Also 3% of compute is so much compute; probably more than the “20% to date over four years” that OpenAI promised and then yanked from superalignment. Take your preferred estimate of lab compute spending, multiply by 3%, and ask yourself whether a rushed unreasonable lab would grant that much money to people working on a topic it didn’t care for, at the expense of those it did.
I think Zac is trying to say they left not to protest, but instead because they didn’t think staying was viable (for whatever research and/or implementation they wanted to do).
On my views (not Zac’s), “staying wouldn’t be viable for someone who was willing to work in a potentially pretty unpleasant work environment and focus on implementation (and currently prepping for this implementation)” doesn’t seem like an accurate description of the situation. (See also Buck’s comment here.)
It does seem to imply that, doesn’t it? I respect the people leaving, and I think it does send a valuable message. And it seems very valuable to have safety-conscious people on the inside.
The question is “are the safety-conscious people effectual at all, and what are their opportunity costs?”.
i.e. are the cheap things they can do that don’t step on anyone’s toes that helpful-on-the-margin, better than what they’d be able to do at another company? (I don’t know the answer, depends on the people).
Not Buck but I think it does unless of course they Saw Something and decided that safety efforts weren’t going to work. The essay seems to hinge on safety people being able to make models safer, which sounds plausible but I’m sure they already knew that. Given their insider information and conclusions about their ability to make a positive impact, then it seems less plausible that their safety efforts would succeed. Maybe whether or not someone has already quit is an indication of how impactful their safety work is. It also varies by lab, with OpenAI having many safety conscious quitters but other labs having much fewer (I want to say none, but maybe I just haven’t heard of any).
The other thing to think about is whether or not people who quit and claimed it was due to safety reasons were being honest about that. I’d like to believe that they were, but all companies have culture/performance expectations that their employees might not want to meet and quitting for safety reasons sounds better than quitting over performance issues.
Does this imply that fewer safety people should quit leading labs to protest poor safety policies?
I’ve talked to a lot of people who have left leading AI companies for reasons related to thinking that their company was being insufficiently cautious. I wouldn’t usually say that they’d left “in protest”; for example, most of them haven’t directly criticized the companies after leaving.
In my experience, the main reason that most of these people left was that they found it very unpleasant to working there and thought their research would be better elsewhere, not that they wanted to protest poor safety policies per se. I usually advise such people against leaving if the company has very few safety staff, but it depends on their skillset.
(These arguments don’t apply to Anthropic: there are many people there who I think will try to implement reasonable safety techniques, so on the current margin, the benefits to safety technique implementation of marginal people seems way lower. It might still make sense to work at Anthropic, especially if you think it’s a good place to do safety research that can be exported.)
Incidentally, I’m happy to talk to people who are considering leaving AI companies and give them much more specific advice.
My impression is that few (one or two?) of the safety people who have quit a leading lab did so to protest poor safety policies, and of those few none saw staying as a viable option.
Relatedly, I think Buck far overestimates the influence and resources of safety-concerned staff in a ‘rushed unreasonable developer’.
While this isn’t amazing evidence, my sense is there have been around 6 people who quit who in-parallel to them announcing their leave called out OpenAI’s reckless attitude towards risk (at various levels of explicitness, but quite strongly in all cases by standard professional norms).
It’s hard to say that people quit “to protest safety policies”, but they definitely used their leaving to protest safety policies. My sense is almost everyone who left in the last year (Daniel, William, Richard, Steven Adler, Miles) did so with a pretty big public message.
Also Rosie Campbell https://x.com/RosieCampbell/status/1863017727063113803
I don’t think Miles’ or Richard’s stated reasons for resigning included safety policies, for example.
But my broader point is that “fewer safety people should quit leading labs to protest poor safety policies” is basically a non-sequitor from “people have quit leading labs because they think they’ll be more effective elsewhere”, whether because they want to do something different or independent, or because they no longer trust the lab to behave responsibly.
Hmm, you have a very different read of Richard’s message than I do. I agree Miles’ statement did not reason through safety policies, but IMO his blogging since then has included a lot of harsh words for OpenAI, in a way that at least to me made the connection clear (and I think also to many others, but IDK, it’s still doing some tea-leaf reading).
FWIW I think of “OpenAI leadership being untrustworthy” (a significant factor in me leaving) as different from “OpenAI having bad safety policies” (not a significant factor in me leaving). Not sure if it matters, I expect that Scott was using “safety policies” more expansively than I do. But just for the sake of clarity:
I am generally pretty sympathetic to the idea that it’s really hard to know what safety policies to put in place right now. Many policies pushed by safety people (including me, in the past) have been mostly kayfabe (e.g. being valuable as costly signals, not on the object level). There are a few object-level safety policies that I really wish OpenAI would do right now (most clearly, implementing better security measures) but I didn’t leave because of that (if I had, I would have tried harder to check before I left what security measures OpenAI did have, made specific objections internally about them before I left, etc).
This may just be a semantic disagreement, it seems very reasonable to define “don’t make employees sign non-disparagements” as a safety policy. But in my mind at least stuff like that is more of a lab governance policy (or maybe a meta-level safety policy).
(I meant the more expansive definition. Plausible that me and Zac talked past each other because of that)
As in, you don’t expect they’ll be able to implement stuff even if it doesn’t make anyone’s workflow harder or you don’t expect they’ll be able to get that much compute?
Naively, we might expect ~1% of compute as we might expect around 1000 researchers and 10/1000 is 1%. Buck said 3% because I argued for increasing this number. My case would be that there will be bunch of cases where the thing they want to do is obviously reasonable and potentially justifiable from multiple perspectives (do some monitoring of internal usage, fine-tune a model for forecasting/advice, use models to do safety research) such that they can pull somewhat more compute than just the head count would suggest.
I also agree with Zac, maybe if you had a really well-selected group of 10 people you could do something, but 10 randomly selected AGI safety researchers probably don’t accomplish much.
By far my biggest objection is that there are approximately zero useful things that “[don’t] make anyone’s workflow harder”. I expect you’re vastly underestimating the complexity of production systems and companies that build them, and the number of constraints they are under. (You are assuming a do-ocracy though, depending on how much of a do-ocracy it is (e.g. willing to ignore laws) I could imagine changing my mind here.)
EDIT: I could imagine doing asynchronous monitoring of internal deployments. This is still going to make some workflows harder, but probably not a ton, so it seems surmountable. Especially since you could combine it with async analyses that the unreasonable developer actually finds useful.
EDIT 2 (Feb 7): To be clear, I also disagree with the compute number. I’m on board with starting with 1% since they are 1% of the headcount. But then I would decrease it first because they’re not a high-compute capabilities team, and second because whatever they are doing should be less useful to the company than whatever the other researchers are doing (otherwise why weren’t the other researchers doing it?), so maybe I’d estimate 0.3%. But this isn’t super cruxy because I think you can do useful safety work with just 0.3% of the compute.
Again, I could imagine getting more compute with a well-selected group of 10 people (though even then 3% seems unlikely, I’m imagining more like 1%), but I don’t see why in this scenario you should assume you get a well-selected group, as opposed to 10 random AGI safety researchers.
Yep, I think that at least some of the 10 would have to have some serious hustle and political savvy that is atypical (but not totally absent) among AI safety people.
What laws are you imagine making it harder to deploy stuff? Notably I’m imagining these people mostly doing stuff with internal deployments.
I think you’re overfixating on the experience of Google, which has more complicated production systems than most.
I agree with Rohin that there are approximately zero useful things that don’t make anyone’s workflow harder. The default state is “only just working means working, so I’ve moved on to the next thing” and if you want to change something there’d better be a benefit to balance the risk of breaking it.
Also 3% of compute is so much compute; probably more than the “20% to date over four years” that OpenAI promised and then yanked from superalignment. Take your preferred estimate of lab compute spending, multiply by 3%, and ask yourself whether a rushed unreasonable lab would grant that much money to people working on a topic it didn’t care for, at the expense of those it did.
Many more than two safety-concerned people have left AI companies for reasons related to thinking that those companies are reckless.
I think Zac is trying to say they left not to protest, but instead because they didn’t think staying was viable (for whatever research and/or implementation they wanted to do).
On my views (not Zac’s), “staying wouldn’t be viable for someone who was willing to work in a potentially pretty unpleasant work environment and focus on implementation (and currently prepping for this implementation)” doesn’t seem like an accurate description of the situation. (See also Buck’s comment here.)
It does seem to imply that, doesn’t it? I respect the people leaving, and I think it does send a valuable message. And it seems very valuable to have safety-conscious people on the inside.
The question is “are the safety-conscious people effectual at all, and what are their opportunity costs?”.
i.e. are the cheap things they can do that don’t step on anyone’s toes that helpful-on-the-margin, better than what they’d be able to do at another company? (I don’t know the answer, depends on the people).
Not Buck but I think it does unless of course they Saw Something and decided that safety efforts weren’t going to work. The essay seems to hinge on safety people being able to make models safer, which sounds plausible but I’m sure they already knew that. Given their insider information and conclusions about their ability to make a positive impact, then it seems less plausible that their safety efforts would succeed. Maybe whether or not someone has already quit is an indication of how impactful their safety work is. It also varies by lab, with OpenAI having many safety conscious quitters but other labs having much fewer (I want to say none, but maybe I just haven’t heard of any).
The other thing to think about is whether or not people who quit and claimed it was due to safety reasons were being honest about that. I’d like to believe that they were, but all companies have culture/performance expectations that their employees might not want to meet and quitting for safety reasons sounds better than quitting over performance issues.