The main examples in my mind are the Mongol conquests and Western colonialism, which I think were the biggest atrocities in history and were more due to power imbalance than fanaticism.
But there’s maybe a more general point I want to make. I think focusing on benevolence isn’t the right path. Let’s say we build a powerful benevolent thing. How can we make sure it stays benevolent to us? After all, value drift is always possible, we don’t have any math to rule it out.
For a while I thought the solution should be some kind of “continuous alignment”, being able to influence the powerful thing as it evolves. But then I realized that it’s simpler than that. Being able to influence a powerful thing that doesn’t want to be influenced is simply another synonym for “having power”. The problem of making sure a powerful thing stays benevolent is exactly the same as the problem of making sure power is spread out, so the powerful thing can be kept in check. The two things mean the same thing.
So now that’s what I’m arguing for in these threads. I want people to get over the framing of “power imbalance is ok as long as the thing is benevolent, so we should focus on ensuring benevolence”, and switch to “power is always subject to value drift, so power imbalance is dangerous in itself, and we should focus on making power spread out”. It feels a really important point to me. Does that make sense?
The main examples in my mind are the Mongol conquests and Western colonialism, which I think were the biggest atrocities in history and were more due to power imbalance than fanaticism.
But there’s maybe a more general point I want to make. I think focusing on benevolence isn’t the right path. Let’s say we build a powerful benevolent thing. How can we make sure it stays benevolent to us? After all, value drift is always possible, we don’t have any math to rule it out.
For a while I thought the solution should be some kind of “continuous alignment”, being able to influence the powerful thing as it evolves. But then I realized that it’s simpler than that. Being able to influence a powerful thing that doesn’t want to be influenced is simply another synonym for “having power”. The problem of making sure a powerful thing stays benevolent is exactly the same as the problem of making sure power is spread out, so the powerful thing can be kept in check. The two things mean the same thing.
So now that’s what I’m arguing for in these threads. I want people to get over the framing of “power imbalance is ok as long as the thing is benevolent, so we should focus on ensuring benevolence”, and switch to “power is always subject to value drift, so power imbalance is dangerous in itself, and we should focus on making power spread out”. It feels a really important point to me. Does that make sense?