I agree with most of the arguments and most of the vision in this post, but I still think the fundamental problem we face is that no one, today, knows how to build a(n AI) system that reliably values any particular chosen thing. We’re getting better, especially in regards to moderately powerful current and near future systems that are meaningfully constrained by the power of other people and systems. But as I understand it this is still a deep, unsolved problem. In other words, when you say:
Historically, you can trace the ebb and flow of the plight of the average person by how decentralizing or centralizing the technology most essential for national power is, and how much that technology creates mutual dependencies that make it hard for the elite to defect against the masses.
I think this dynamic is much deeper than the impression I got from this post suggests.
Often, this then leads to calls for centralization, from Oppenheimer advocating for world government in response to the atomic bomb, to Nick Bostrom’s proposal that comprehensive surveillance and totalitarian world government might be required to prevent existential risk from destructive future technologies.
Was Oppenheimer wrong? AFAICT we did, in fact, build a (fairly competent by human standards) limited form of world government for the specific goal of constraining access to nuclear weapons. The US and USSR seized overwhelming power in regards to nukes just about as soon as they were able to do so, and then conspired to deny anyone else from acquiring large amounts of that same power. In the process they altered (and slowed, and in some ways crippled) the potential for nuclear technology to solve civilian problems, most notably in energy. They did so for preservation of themselves and the world, so yay, but they did do it. In the process they had to waste a lot of resources that could in principle have been used to do much more valuable things, had they felt safe to do so.
I agree with most of the arguments and most of the vision in this post, but I still think the fundamental problem we face is that no one, today, knows how to build a(n AI) system that reliably values any particular chosen thing. We’re getting better, especially in regards to moderately powerful current and near future systems that are meaningfully constrained by the power of other people and systems. But as I understand it this is still a deep, unsolved problem. In other words, when you say:
I think this dynamic is much deeper than the impression I got from this post suggests.
Was Oppenheimer wrong? AFAICT we did, in fact, build a (fairly competent by human standards) limited form of world government for the specific goal of constraining access to nuclear weapons. The US and USSR seized overwhelming power in regards to nukes just about as soon as they were able to do so, and then conspired to deny anyone else from acquiring large amounts of that same power. In the process they altered (and slowed, and in some ways crippled) the potential for nuclear technology to solve civilian problems, most notably in energy. They did so for preservation of themselves and the world, so yay, but they did do it. In the process they had to waste a lot of resources that could in principle have been used to do much more valuable things, had they felt safe to do so.