I know a person (who is not me) who was going to college to eventually work on AI alignment, wholeheartedly absorbed this community’s “we need to work at full speed at all times or we’re all gonna die” attitude and took on far too much work, had a complete mental breakdown, and is now a depressed nihilist hedonist.
The problem you speak of is not hypothetical. This community’s doomerism and chronic stress has already incurred real costs in brain drain.
Skepticism about the alignment of government and the incentives thereof has existed for almost as long as governments have. Elections, for example, are a crude but better-than-nothing attempt to align political interests with public interests, and much ink has been spilled on the subject of improving this alignment and even whether alignment to the general public opinion is a good idea.
Far less such discussion has occurred in the case of extremely large companies, as they are a relatively newer concept.
The fact that there is not really an equivalent for AGI is admittedly a place where this analogy breaks down.
A pithier version of this has been suggested to me as “[Corporations are] like paperclippers except for money”.
It may also be worth noting how a sufficiently advanced “algorithm” could start making its own “decisions”; for example, a search/display algorithm that has been built to maximize advertisement revenue, if given enough resources and no moral boundaries, may suppress search results that contain negative opinions on itself, promote taking down competitors, and/or preferentially display news and arguments that are in favor of allowing Algorithms more power. Skepticism about The Algorithm is a cause many political parties are already able to agree on; the possibility of The Algorithm going FOOM might accelerate public discussions about the development of AI in general.
If we want aligned AI, the people and entities making the AI must also be aligned. Publicly traded corporations are misaligned, due to pressures such as executive bonus pay and the doctrine of fiduciary duty to shareholders.