If the aggregate agency of life on earth doesn’t have the coordination sufficient to avoid it, perhaps. But it seems to me that centralization-first plans don’t have a way to guarantee the agency of the people at the edges that can be strongly durable. I’d hope to design an AI that is structurally inclined to find ways to give the guarantees you’re wanting that doesn’t need to be a ruler to give those guarantees—for example, offering a copy of itself to everyone, being strongly auditable, and allowing people to link up into mesh coordination networks that can peer-reinforce against moloch.
If the aggregate agency of life on earth doesn’t have the coordination sufficient to avoid it, perhaps. But it seems to me that centralization-first plans don’t have a way to guarantee the agency of the people at the edges that can be strongly durable. I’d hope to design an AI that is structurally inclined to find ways to give the guarantees you’re wanting that doesn’t need to be a ruler to give those guarantees—for example, offering a copy of itself to everyone, being strongly auditable, and allowing people to link up into mesh coordination networks that can peer-reinforce against moloch.