selective hiring is very real. lots of people who are xrisk pilled just refuse to join oai. people who care a lot often end up very stressed and leave in large part because of the stress.
the vast majority of people at oai do not think of xrisk from agi as a serious thing. but then again probably a majority dont really truly think of agi as a serious thing.
people absolutely do argue “well if i didn’t do it, someone else would. and even if oai stopped, some other company would do it” to justify their work.
compartmentalization is probably not a big part of the reason, at least not yet. historically things don’t get compartmentalized often, and even when they do, i don’t think it makes the difference between being worried and not being worried about xrisk for that many people
as companies get big, teams A B C not talking to each other is the default order of the world and it takes increasing effort to get them to talk to each other. and even getting them talking is not enough to change their courses of action, which often requires a lot of work from higher up. this hampers everything; this is in general why big companies have so many overlapping/redundant teams
people get promoted / allocated more resources if they do things that are obviously useful for the company, as opposed to less obviously useful for the company (i mean, as a company, you kind of understandably have to do this or else die of resource misallocation).
i think quite a few people, especially more senior people, are no longer driven by financial gain. these things are sometimes “i really want to accomplish something great in the field of ML” or “i like writing code” or “i like being part of something important / shaping the future”. my guess is anyone super competent who cares primarily about money quits after a few years and, depending on the concavity of their utility function, either retires on a beach, or founds a startup and raises a gazillion dollars from VCs
it’s pretty difficult to do weird abstract bullshit that doesn’t obviously tie into some kind of real world use case (or fit into the internally-accepted research roadmap to AGI). this has imo hampered both alignment and capabilities. it makes a lot of sense though, like, bell labs didn’t capture most of the value that bell labs created, and academia is the place where weird abstract bullshit is supposed to live, and we’re in some sense quite lucky that industry is willing to fund any of it at all
concretely this means anything alignmenty gets a huge boost if you can argue that it will (a) improve capabilities or (b) prevent some kind of embarrassing safety failure in the model we’re currently serving to gazillions of people. the kinds of things people choose to work on are strongly shaped by this as a result, and probably explains why so much work keeps taking alignment words and using them to mean aligning GPT-5 rather than AGI.
aside from the leopold situation, which had pretty complicated circumstances, people don’t really get fired for caring about xrisk. the few incidents are hard to interpret because of strong confounding factors and could be argued either way. but it’s not so far from base rates so i don’t feel like it’s a huge thing.
my guess is a lot of antipathy towards safety comes from broader antipathy against safetyism as a whole in society, which honestly i (and many people in alignment) have to admit some sympathy towards.
I am not surprised to hear this but also, this is insane.
All the lab heads are repeatedly publicly claiming they could cause human extinction, superintelligence is within reach, and a majority of people at their own labs don’t take them seriously on this.
I’m somewhat confused what causes a group of people who talk to each other everyday, work on the same projects, observe the same evidence, etc to come to such wildly different conclusions about the work they’re doing together and then be uninterested in resolving the disagreement.
Is there a taboo being enforced against discussing these disagreements inside the labs?
this is pretty normal? it’s really hard for leadership to make employees care about or believe specific things. do you really think the average Amazon employee or whatever has strong opinions on the future of delivery drones? does the average Waymo employee have extremely strong beliefs about the future of self driving?
for most people in the world, their job is just a job. people obviously avoid working on things they believe are completely doomed, and tend to work on cool trendy things. but generally most people do not really have strong beliefs about where the stuff they’re working on is going.
no specific taboo is required to ensure that people don’t really iron out deep philosophical disagreements with their coworkers. people care about all sorts of other things in life. they care about money, they care whether they’re enjoying the work, they care whether their coworkers are pleasant to be around, they care about their wife and kids and house.
once you have a company with more than 10 people, it requires constant effort to maintain culture. hiring is way harder if you can only hire people who are aligned, or if you insist on aligning people. if you grow very fast (and openai has grown very fast—it’s approximately doubled every single year I’ve been here), it’s inevitable that your culture will splinter. forget about having everyone on the same page; you’re going to have entire little googletowns and amazontowns and so on of people who bring Google or Amazon culture with them and agglomerate with other recent transplants from those companies.
do you really think the average Amazon employee or whatever has strong opinions on the future of delivery drones? does the average Waymo employee have extremely strong beliefs about the future of self driving?
I assume yes a lot of employees do believe technical capabilities are further ahead of what the rest of the public is aware of, because they have insider knowledge. And they are aware they are accelerating it.
They might not deeply care about drones but they are aware what is going on.
With AI, you can’t not care about ASI or extinction or catastrophe or dictatorship or any of the words thrown around, as they directly affect your life too.
once you have a company with more than 10 people, it requires constant effort to maintain culture. hiring is way harder if you can only hire people who are aligned, or if you insist on aligning people.
Then I expect OpenAI to splinter hard as we get closer to ASI.
I think it’s notably abnormal specifically because it wasn’t the “default” equilibrium for OpenAI specifically.
Like earlier you mentioned:
selective hiring is very real. lots of people who are xrisk pilled just refuse to join oai. people who care a lot often end up very stressed and leave in large part because of the stress.
and
most cases of xrisk people leaving are just because people felt sidelined/unhappy and chose to leave.
One model of this is “its normal that people at any company don’t have strong opinions about their work”. Another model is “lots of people in various positions did in fact have strong opinions about this given the stakes and left”.
If you send such strong signals about safety that people preemptively filter out of the hiring pipeline, then people who are already there with strong opinions on safety feel sidelined, IMO the obvious interpretation is “you actively filtered against people with strong views on safety”.
Can you gesture at what you are basing this guess on?
Ignore this if you are busy, but I was also wondering: Do you have any takes on “do people get fired over acting on beliefs about xrisk”, as opposed to “people getting fired over caring about xrisk”? And perhaps on “whether people would get fired over acting on xrisk beliefs, so they don’t act on them”? (Though this seems difficult to operationalise.)
as far as I’m aware, the only person who can be argued to have ever been fired for acting on beliefs about x risk is leopold, and the circumstances there are pretty complicated. since I don’t think he’s the only person to have ever acted on xrisk at oai to the extent he did, I don’t think this is just because other people don’t do anything about xrisk.
most cases of xrisk people leaving are just because people felt sidelined/unhappy and chose to leave. which is ofc also bad, but quite different.
I don’t think this is just because other people don’t do anything about xrisk.
Why is that? If someone went off and consistently worked on an agenda that was directly xrisk related (that didn’t contribute to a short term capabilities or product safety) you’re saying they wouldn’t get sidelined / not allocated resources / fired?
my guess:
selective hiring is very real. lots of people who are xrisk pilled just refuse to join oai. people who care a lot often end up very stressed and leave in large part because of the stress.
the vast majority of people at oai do not think of xrisk from agi as a serious thing. but then again probably a majority dont really truly think of agi as a serious thing.
people absolutely do argue “well if i didn’t do it, someone else would. and even if oai stopped, some other company would do it” to justify their work.
compartmentalization is probably not a big part of the reason, at least not yet. historically things don’t get compartmentalized often, and even when they do, i don’t think it makes the difference between being worried and not being worried about xrisk for that many people
as companies get big, teams A B C not talking to each other is the default order of the world and it takes increasing effort to get them to talk to each other. and even getting them talking is not enough to change their courses of action, which often requires a lot of work from higher up. this hampers everything; this is in general why big companies have so many overlapping/redundant teams
people get promoted / allocated more resources if they do things that are obviously useful for the company, as opposed to less obviously useful for the company (i mean, as a company, you kind of understandably have to do this or else die of resource misallocation).
i think quite a few people, especially more senior people, are no longer driven by financial gain. these things are sometimes “i really want to accomplish something great in the field of ML” or “i like writing code” or “i like being part of something important / shaping the future”. my guess is anyone super competent who cares primarily about money quits after a few years and, depending on the concavity of their utility function, either retires on a beach, or founds a startup and raises a gazillion dollars from VCs
it’s pretty difficult to do weird abstract bullshit that doesn’t obviously tie into some kind of real world use case (or fit into the internally-accepted research roadmap to AGI). this has imo hampered both alignment and capabilities. it makes a lot of sense though, like, bell labs didn’t capture most of the value that bell labs created, and academia is the place where weird abstract bullshit is supposed to live, and we’re in some sense quite lucky that industry is willing to fund any of it at all
concretely this means anything alignmenty gets a huge boost if you can argue that it will (a) improve capabilities or (b) prevent some kind of embarrassing safety failure in the model we’re currently serving to gazillions of people. the kinds of things people choose to work on are strongly shaped by this as a result, and probably explains why so much work keeps taking alignment words and using them to mean aligning GPT-5 rather than AGI.
aside from the leopold situation, which had pretty complicated circumstances, people don’t really get fired for caring about xrisk. the few incidents are hard to interpret because of strong confounding factors and could be argued either way. but it’s not so far from base rates so i don’t feel like it’s a huge thing.
my guess is a lot of antipathy towards safety comes from broader antipathy against safetyism as a whole in society, which honestly i (and many people in alignment) have to admit some sympathy towards.
I am not surprised to hear this but also, this is insane.
All the lab heads are repeatedly publicly claiming they could cause human extinction, superintelligence is within reach, and a majority of people at their own labs don’t take them seriously on this.
I’m somewhat confused what causes a group of people who talk to each other everyday, work on the same projects, observe the same evidence, etc to come to such wildly different conclusions about the work they’re doing together and then be uninterested in resolving the disagreement.
Is there a taboo being enforced against discussing these disagreements inside the labs?
this is pretty normal? it’s really hard for leadership to make employees care about or believe specific things. do you really think the average Amazon employee or whatever has strong opinions on the future of delivery drones? does the average Waymo employee have extremely strong beliefs about the future of self driving?
for most people in the world, their job is just a job. people obviously avoid working on things they believe are completely doomed, and tend to work on cool trendy things. but generally most people do not really have strong beliefs about where the stuff they’re working on is going.
no specific taboo is required to ensure that people don’t really iron out deep philosophical disagreements with their coworkers. people care about all sorts of other things in life. they care about money, they care whether they’re enjoying the work, they care whether their coworkers are pleasant to be around, they care about their wife and kids and house.
once you have a company with more than 10 people, it requires constant effort to maintain culture. hiring is way harder if you can only hire people who are aligned, or if you insist on aligning people. if you grow very fast (and openai has grown very fast—it’s approximately doubled every single year I’ve been here), it’s inevitable that your culture will splinter. forget about having everyone on the same page; you’re going to have entire little googletowns and amazontowns and so on of people who bring Google or Amazon culture with them and agglomerate with other recent transplants from those companies.
I appreciate your answer, it adds useful info.
I assume yes a lot of employees do believe technical capabilities are further ahead of what the rest of the public is aware of, because they have insider knowledge. And they are aware they are accelerating it.
They might not deeply care about drones but they are aware what is going on.
With AI, you can’t not care about ASI or extinction or catastrophe or dictatorship or any of the words thrown around, as they directly affect your life too.
Then I expect OpenAI to splinter hard as we get closer to ASI.
I think it’s notably abnormal specifically because it wasn’t the “default” equilibrium for OpenAI specifically.
Like earlier you mentioned:
and
One model of this is “its normal that people at any company don’t have strong opinions about their work”. Another model is “lots of people in various positions did in fact have strong opinions about this given the stakes and left”.
If you send such strong signals about safety that people preemptively filter out of the hiring pipeline, then people who are already there with strong opinions on safety feel sidelined, IMO the obvious interpretation is “you actively filtered against people with strong views on safety”.
Can you gesture at what you are basing this guess on?
Ignore this if you are busy, but I was also wondering: Do you have any takes on “do people get fired over acting on beliefs about xrisk”, as opposed to “people getting fired over caring about xrisk”?
And perhaps on “whether people would get fired over acting on xrisk beliefs, so they don’t act on them”? (Though this seems difficult to operationalise.)
as far as I’m aware, the only person who can be argued to have ever been fired for acting on beliefs about x risk is leopold, and the circumstances there are pretty complicated. since I don’t think he’s the only person to have ever acted on xrisk at oai to the extent he did, I don’t think this is just because other people don’t do anything about xrisk.
most cases of xrisk people leaving are just because people felt sidelined/unhappy and chose to leave. which is ofc also bad, but quite different.
Why is that? If someone went off and consistently worked on an agenda that was directly xrisk related (that didn’t contribute to a short term capabilities or product safety) you’re saying they wouldn’t get sidelined / not allocated resources / fired?
Looks like you accidentally dropped a sentence there.