Another dynamic is that employees at AI companies often think that their AI company not going bankrupt will allow P(doom) reduction, at least via the classic mechanism of “having more power lets us do more good things” (e.g. advocating for good policies using the clout of a leading AI company, doing the cheap and important safety/security stuff that the counterfactual company would not even bother doing—and demonstrating that these cheap and important things are cheap and important, using AIs in net-good ways that other companies would have not bothered doing, not trying to spread dangerous ideologies, …).
And it’s unclear how incorrect this reasoning is. This seems to me like a sensible thing to do if you know the company will actually use its power for good—but it’s worrisome that so many people in so many different AI companies think they are in an AI company that will do something better with the power than the company that would replace them if they went bankrupt.
I think that “the company is doing sth first-order risky but employees think it’s actually good because going bankrupt would shift power towards some other more irresponsible actors” is at least as central as “the company is doing something risky to avoid going bankrupt and employees would prefer that the company would not do it, but [dynamic] prevents employees from stopping it”.
Another dynamic is that employees at AI companies often think that their AI company not going bankrupt will allow P(doom) reduction, at least via the classic mechanism of “having more power lets us do more good things” (e.g. advocating for good policies using the clout of a leading AI company, doing the cheap and important safety/security stuff that the counterfactual company would not even bother doing—and demonstrating that these cheap and important things are cheap and important, using AIs in net-good ways that other companies would have not bothered doing, not trying to spread dangerous ideologies, …).
And it’s unclear how incorrect this reasoning is. This seems to me like a sensible thing to do if you know the company will actually use its power for good—but it’s worrisome that so many people in so many different AI companies think they are in an AI company that will do something better with the power than the company that would replace them if they went bankrupt.
I think that “the company is doing sth first-order risky but employees think it’s actually good because going bankrupt would shift power towards some other more irresponsible actors” is at least as central as “the company is doing something risky to avoid going bankrupt and employees would prefer that the company would not do it, but [dynamic] prevents employees from stopping it”.