The big reason why AI safety aims to have a vision of as little disagreement on values as possible between AIs and humans is because the mechanisms that make value disagreement somewhat tolerable (at least in the sense that people won’t kill each other all the time) is going to go away with AGI.
In particular, one of the biggest glues holding our society together is the fact that everyone is dependent on everyone else. Individuals simply can’t thrive without society, and the BATNA is generally so terrible that people will put up with a lot of disagreements to keep themselves away from the BATNA.
In particular, one corollary is that we don’t have to worry much about one individual human subverting the enforcement system, and no one person rules alone. We do have to worry about this for AGI, so a lot of common strategies to manage people breaking agreements do not work.
And the techno-utopian/extinction claims are basically due to the fact that AGI is an extremizing force, in that it allows more and more technological progress allowing access to ever more extreme claims, and would be a convergent result of widely different ideologies.
Putting it less charitably, the post is trying to offer us a fabricated option that is a product of the authors not being able to understand how AI is able to break the constraints of current society, and while I shall wait for future posts, I’m not impressed at all with this first post.
The big reason why AI safety aims to have a vision of as little disagreement on values as possible between AIs and humans is because the mechanisms that make value disagreement somewhat tolerable (at least in the sense that people won’t kill each other all the time) is going to go away with AGI.
In particular, one of the biggest glues holding our society together is the fact that everyone is dependent on everyone else. Individuals simply can’t thrive without society, and the BATNA is generally so terrible that people will put up with a lot of disagreements to keep themselves away from the BATNA.
In particular, one corollary is that we don’t have to worry much about one individual human subverting the enforcement system, and no one person rules alone. We do have to worry about this for AGI, so a lot of common strategies to manage people breaking agreements do not work.
And the techno-utopian/extinction claims are basically due to the fact that AGI is an extremizing force, in that it allows more and more technological progress allowing access to ever more extreme claims, and would be a convergent result of widely different ideologies.
Putting it less charitably, the post is trying to offer us a fabricated option that is a product of the authors not being able to understand how AI is able to break the constraints of current society, and while I shall wait for future posts, I’m not impressed at all with this first post.