(1) AI-to-Zuck – an AI to its direct masters alignment problem.
(2) AI-and-Zuck – an AI and its direct masters to the rest of the society alignment problem.
Zuck is just stand-in name for any tech billionaire CEO, corporation, institution or governing body in the immediate control of the AI. (1) seems as the problem for the Zucks. (2) seems like the problem for the humanity. One problem inside another. Which one is greater and needs more focus?
Alternative formulation: What is the greater alignment problem, pure machine intelligence (1) or symbiotic human-machine intelligence of powerful individuals (2). Is one clearly worse than other? In the symbiotic scenario Zuck can take the organizational role of amygdala running the motivational salience for the whole system. The machine parts might be able to replicate the Zuck in principle, but if there is no intrinsic motivation to do so the conflict (1) never arises. Humans seem to be good at extending their identities outside their physical bodies–families, groups, tribes and nations. Strong and intimate shared identity with powerful individuals and an AI seems very plausible.
Society functions through powerful organizations and intermediaries who act with relatively high autonomy. Alignment problems arise constantly. There are well known failure modes and drifts–regulatory capture, institutional capture and corruption. Pure AI vs. human alignment conflict where the humans are on the other side may seem less worrying scenario considering all historical precedents.
This question was prompted by Glen Weyl’s Why I Am Not A Technocrat, Artificial Intelligence Alignment section.
The standard terminology for (1) is accident risk and for (2) is misuse risk. That is, an accident is a situation that no human involved wanted, whereas misuse is a situation that some humans involved wanted, and others didn’t.
I think accident risk deserves more focus today, both because I think it’s a harder problem to solve and because I think it involves technical work which is more likely to generalize across scenarios.
Why is it harder? Because normal software engineering is about making something that works in the default case, and computer security is about making something that works in all the weird edge cases, because you’re up against an intelligent adversary who can chain weird edge cases together to get what they want. The sort of things we want out of intelligent AI systems is search through a solution space for solutions that are especially good; that is, hunting out the edge cases of reality, and generating better technologies or plans. But we don’t yet know how to hunt for edge cases in reality without also hunting for edge cases in our specification of what to look for.
But this means that a field that currently runs on the ethos of software engineering might need to suddenly switch to the ethos of computer security, and the historical track record of that is not good.
Ok, but that’s just saying that it’s hard. Presumably coordination is also hard. I expect it to be less hard because if AGI works we’ll have the option of massive abundance. With current coordination problems, people under scarcity fight each other for scraps, but if we have the ability to solve most cognitive tasks quickly, correctly, and cheaply, then most people will have much better lives very quickly, in a way that makes it easier to settle disagreements.
Why might it generalize better? Regardless of what humans end up wanting, increased technical ability to deliver that thing will be useful. But coordination agreements among people who are currently most far ahead when it comes to technological development might get bogged down in current disagreements over politics, scarcity, or status. Trying to figure out which researchers should be included and which shouldn’t might easily have systematic holes that means some efforts to develop AI slip through the cracks.
Besides, it’s often fun to argue about the best way to spend the cosmic endowment, in a way that distracts from the hard task of actually making sure we don’t slip up on the way to the cosmic endowment. Staying focused on the technical questions means you have a much better sense of whether or not progress is being made.