The dire part of alignment is that we know that most human beings themselves are not internally aligned, but they become aligned only because they benefits from living in communities. And in general, most organisms by themselves are “non-aligned”, if you allow me to bend the term to indicate anything that might consume/expand its environment to maximize some internal reward function.
But all biological organisms are embodied and have strong physical limits, so most organisms become part of self-balancing ecosystems.
AGI, being an un-embodied agent, doesn’t have strong physical limits in its capabilities so it is hard to see how it/they could find advantageous or would they be forced to cooperate.
The dire part of alignment is that we know that most human beings themselves are not internally aligned, but they become aligned only because they benefits from living in communities. And in general, most organisms by themselves are “non-aligned”, if you allow me to bend the term to indicate anything that might consume/expand its environment to maximize some internal reward function.
But all biological organisms are embodied and have strong physical limits, so most organisms become part of self-balancing ecosystems.
AGI, being an un-embodied agent, doesn’t have strong physical limits in its capabilities so it is hard to see how it/they could find advantageous or would they be forced to cooperate.