Similarly, our ability to coordinate through language also plays a huge role in explaining our power compared to other animals. But, on a first approximation, other animals can’t coordinate at all, making this distinction much less impressive. The first AGIs we construct will be born into a culture already capable of coordinating, and sharing knowledge, making the potential power difference between AGI and humans relatively much smaller than between humans and other animals, at least at first.
I basically buy the story that human intelligence is less useful that human coordination; i.e. it’s the intelligence of “humanity” the entity that matters, with the intelligence of individual humans relevant only as, like, subcomponents of that entity.
But… shouldn’t this mean you expect AGI civilization to totally dominate human civilization? They can read each other’s source code, and thus trust much more deeply! They can transmit information between them at immense bandwidths! They can clone their minds and directly learn from each other’s experiences!
Like, one scenario I visualize a lot is the NHS having a single ‘DocBot’, i.e. an artificial doctor run on datacenters that provides medical advice and decision-making for everyone in the UK (while still working with nurses and maybe surgeons and so on). Normally I focus on the way that it gets about three centuries of experience treating human patients per day, but imagine the difference in coordination capacity between DocBot and the BMA.
Having seen undeniable, large economic effects from AI, policymakers will eventually realize that AGI is important, and will launch massive efforts to regulate it.
I think everyone expects this, and often disagree on the timescale on which it will arrive. See, for example, Elon Musk’s speech to the US National Governors Association, where he argues that the reactive regulation model will be too slow to handle the crisis.
But I think the even more important disagreement is on whether or not regulations should be expected to work. Ok, so you make it so that only corporations with large compliance departments can run AGI. How does that help? There was a tweet by Matt Yglesias a while ago that I can’t find now, which went something like: “a lot of smart people are worried about AI, and when you ask them what the government can do about it, they have no idea; this is an extremely wild situation from the perspective of a policy person.” A law that says “don’t run the bad code” is predicated on the ability to tell the good code from the bad code, which is the main thing we’re missing and don’t know how to get!
And if you say something like “ok, one major self-driving car accident will be enough to convince everyone to do the Butlerian Jihad and smash all the computers”, that’s really not how it looks to me. Like, the experience of COVID seems a lot like “people who were doing risky research in labs got out in front of everyone else to claim that the lab leak hypothesis was terrible and unscientific, and all of the anti-disinformation machinery was launched to suppress it, and it took a shockingly long time to even be able to raise the hypothesis, and it hasn’t clearly swept the field, and legislation to do something about risky research seems like it definitely isn’t a slam dunk.”
When we get some AI warning signs, I expect there are going to be people with the ability to generate pro-AI disinfo and a strong incentive to do so. I expect there to be significant latent political polarization which will tangle up any attempt to do something useful about it. I expect there won’t be anything like the international coordination that was necessary to set up anti-nuclear-proliferation efforts to set up the probably harder problem of anti-AGI-proliferation efforts.
But… shouldn’t this mean you expect AGI civilization to totally dominate human civilization? They can read each other’s source code, and thus trust much more deeply! They can transmit information between them at immense bandwidths! They can clone their minds and directly learn from each other’s experiences!
This is 100% correct, and part of why I expect the focus on superintelligence, while literally true, is bad for AI outreach. There’s a much simpler (and empirically, in my experience, more convincing) explanation of why we lose to even an AI with an IQ of 110. It is Dath Ilan, and we are Earth. Coordination is difficult for humans and the easy part for AIs.
I will note that Eliezer wrote That Alien Message a long time ago I think in part to try to convey the issue to this perspective, but it’s mostly about “information-theoretic bounds are probably not going to be tight” in a simulation-y universe instead of “here’s what coordination between computers looks like today”. I do predict the coordination point would be good to include in more of the intro materials.
But… shouldn’t this mean you expect AGI civilization to totally dominate human civilization? They can read each other’s source code, and thus trust much more deeply! They can transmit information between them at immense bandwidths! They can clone their minds and directly learn from each other’s experiences!
I don’t think it’s obvious that this means that AGI is more dangerous, because it means that for a fixed total impact of AGI, the AGI doesn’t have to be as competent at individual thinking (because it leans relatively more on group thinking). And so at the point where the AGIs are becoming very powerful in aggregate, this argument pushes us away from thinking they’re good at individual thinking.
Also, it’s not obvious that early AIs will actually be able to do this if their creators don’t find a way to train them to have this affordance. ML doesn’t currently normally make AIs which can helpfully share mind-states, and it probably requires non-trivial effort to hook them up correctly to be able to share mind-state.
They can read each other’s source code, and thus trust much more deeply!
Being able to read source code doesn’t automatically increase trust—you also have to be able to verify that the code being shared with you actually governs the AGI’s behavior, despite that AGI’s incentives and abilities to fool you.
(Conditional on the AGIs having strongly aligned goals with each other, sure, this degree of transparency would help them with pure coordination problems.)
I basically buy the story that human intelligence is less useful that human coordination; i.e. it’s the intelligence of “humanity” the entity that matters, with the intelligence of individual humans relevant only as, like, subcomponents of that entity.
But… shouldn’t this mean you expect AGI civilization to totally dominate human civilization? They can read each other’s source code, and thus trust much more deeply! They can transmit information between them at immense bandwidths! They can clone their minds and directly learn from each other’s experiences!
Like, one scenario I visualize a lot is the NHS having a single ‘DocBot’, i.e. an artificial doctor run on datacenters that provides medical advice and decision-making for everyone in the UK (while still working with nurses and maybe surgeons and so on). Normally I focus on the way that it gets about three centuries of experience treating human patients per day, but imagine the difference in coordination capacity between DocBot and the BMA.
I think everyone expects this, and often disagree on the timescale on which it will arrive. See, for example, Elon Musk’s speech to the US National Governors Association, where he argues that the reactive regulation model will be too slow to handle the crisis.
But I think the even more important disagreement is on whether or not regulations should be expected to work. Ok, so you make it so that only corporations with large compliance departments can run AGI. How does that help? There was a tweet by Matt Yglesias a while ago that I can’t find now, which went something like: “a lot of smart people are worried about AI, and when you ask them what the government can do about it, they have no idea; this is an extremely wild situation from the perspective of a policy person.” A law that says “don’t run the bad code” is predicated on the ability to tell the good code from the bad code, which is the main thing we’re missing and don’t know how to get!
And if you say something like “ok, one major self-driving car accident will be enough to convince everyone to do the Butlerian Jihad and smash all the computers”, that’s really not how it looks to me. Like, the experience of COVID seems a lot like “people who were doing risky research in labs got out in front of everyone else to claim that the lab leak hypothesis was terrible and unscientific, and all of the anti-disinformation machinery was launched to suppress it, and it took a shockingly long time to even be able to raise the hypothesis, and it hasn’t clearly swept the field, and legislation to do something about risky research seems like it definitely isn’t a slam dunk.”
When we get some AI warning signs, I expect there are going to be people with the ability to generate pro-AI disinfo and a strong incentive to do so. I expect there to be significant latent political polarization which will tangle up any attempt to do something useful about it. I expect there won’t be anything like the international coordination that was necessary to set up anti-nuclear-proliferation efforts to set up the probably harder problem of anti-AGI-proliferation efforts.
This is 100% correct, and part of why I expect the focus on superintelligence, while literally true, is bad for AI outreach. There’s a much simpler (and empirically, in my experience, more convincing) explanation of why we lose to even an AI with an IQ of 110. It is Dath Ilan, and we are Earth. Coordination is difficult for humans and the easy part for AIs.
I will note that Eliezer wrote That Alien Message a long time ago I think in part to try to convey the issue to this perspective, but it’s mostly about “information-theoretic bounds are probably not going to be tight” in a simulation-y universe instead of “here’s what coordination between computers looks like today”. I do predict the coordination point would be good to include in more of the intro materials.
I don’t think it’s obvious that this means that AGI is more dangerous, because it means that for a fixed total impact of AGI, the AGI doesn’t have to be as competent at individual thinking (because it leans relatively more on group thinking). And so at the point where the AGIs are becoming very powerful in aggregate, this argument pushes us away from thinking they’re good at individual thinking.
Also, it’s not obvious that early AIs will actually be able to do this if their creators don’t find a way to train them to have this affordance. ML doesn’t currently normally make AIs which can helpfully share mind-states, and it probably requires non-trivial effort to hook them up correctly to be able to share mind-state.
Being able to read source code doesn’t automatically increase trust—you also have to be able to verify that the code being shared with you actually governs the AGI’s behavior, despite that AGI’s incentives and abilities to fool you.
(Conditional on the AGIs having strongly aligned goals with each other, sure, this degree of transparency would help them with pure coordination problems.)