A lot of important warnings in this post. “Capabilities generalize further than alignment once capabilities start to generalize far” was novel to me and seems very important if true.
I don’t really understand the emphasis on “pivotal acts”, though; there seems to be tons of weak pivotal acts, e.g. ways in which narrow AI or barely-above-human-AGI could help coordinate a global emergency regulatory response by the AI superpowers. Still might be worth focusing our effort on the future worlds where no weak pivotal acts are available, but important to point out this is not the median world.
I could coordinate world superpowers if they wanted to coordinate and were willing to do that. It’s not an intelligence problem, unless the solution is mind-control, and then that’s not a weak pivotal act, it’s an AGI powerful enough to kill you if misaligned.
Mind control is too extreme; I think world superpowers could be coordinated with levels of persuasion greater than one Eliezer but short of mind control. E.g. people are already building narrow persuasion AI capable of generating arguments that are highly persuasive for specific people. A substantially-superhuman but still narrow version of such an AI will very likely be built in the next 5 years, and could be used in a variety of weak pivotal acts (not even in a manipulative way! even a public demonstration of such an AI would make a strong case for coordination, comparable to various weapons treaties).
A lot of important warnings in this post. “Capabilities generalize further than alignment once capabilities start to generalize far” was novel to me and seems very important if true.
I don’t really understand the emphasis on “pivotal acts”, though; there seems to be tons of weak pivotal acts, e.g. ways in which narrow AI or barely-above-human-AGI could help coordinate a global emergency regulatory response by the AI superpowers. Still might be worth focusing our effort on the future worlds where no weak pivotal acts are available, but important to point out this is not the median world.
I could coordinate world superpowers if they wanted to coordinate and were willing to do that. It’s not an intelligence problem, unless the solution is mind-control, and then that’s not a weak pivotal act, it’s an AGI powerful enough to kill you if misaligned.
Mind control is too extreme; I think world superpowers could be coordinated with levels of persuasion greater than one Eliezer but short of mind control. E.g. people are already building narrow persuasion AI capable of generating arguments that are highly persuasive for specific people. A substantially-superhuman but still narrow version of such an AI will very likely be built in the next 5 years, and could be used in a variety of weak pivotal acts (not even in a manipulative way! even a public demonstration of such an AI would make a strong case for coordination, comparable to various weapons treaties).