Michael Nielsen has a thoughtful take on this, as excerpted here and originally published as a postscript here.
He describes how alignment work has often worsened risk to humanity by making products more palatable and salable, which has boosted investment, sped up capability advancements, and thereby increased destructive potential.
Nielsen next states that, even if you think technical safety and alignment work are helpful, it’s still a bad idea to do that work for companies developing frontier AI. As he puts it,
As far as I can tell, at the margin it almost never makes sense to work on market-supplied safety. Capitalism is an incredibly powerful force, and for better and for worse the world is always well-supplied with people willing to do what capital wants. Insofar as alignment is (mostly) a form of market-supplied safety, at the margin it’s more impactful to work on other things. So my current heuristic, and I expect this to be true for quite some time: work on non-market safety, and insofar as you can, avoid doing what the market wants. That means working on governance, it means pause or slowdown, it means new ideas for institutions to govern technology. It mostly doesn’t mean alignment.
I think Haiku’s comment has some great suggestions along these lines.
I’ll add that, at a frontier AI company, it could be very difficult for you to tell if you were helping or making things worse. I haven’t worked at a frontier AI company (and I wouldn’t!), so I don’t speak from direct experience. But in many areas of business, employees are encouraged to feel that their safety concerns are positively influencing company actions, when actual influence is negligible or counterproductive.
Michael Nielsen has a thoughtful take on this, as excerpted here and originally published as a postscript here.
He describes how alignment work has often worsened risk to humanity by making products more palatable and salable, which has boosted investment, sped up capability advancements, and thereby increased destructive potential.
Nielsen next states that, even if you think technical safety and alignment work are helpful, it’s still a bad idea to do that work for companies developing frontier AI. As he puts it,
I think Haiku’s comment has some great suggestions along these lines.
I’ll add that, at a frontier AI company, it could be very difficult for you to tell if you were helping or making things worse. I haven’t worked at a frontier AI company (and I wouldn’t!), so I don’t speak from direct experience. But in many areas of business, employees are encouraged to feel that their safety concerns are positively influencing company actions, when actual influence is negligible or counterproductive.