But… shouldn’t this mean you expect AGI civilization to totally dominate human civilization? They can read each other’s source code, and thus trust much more deeply! They can transmit information between them at immense bandwidths! They can clone their minds and directly learn from each other’s experiences!
This is 100% correct, and part of why I expect the focus on superintelligence, while literally true, is bad for AI outreach. There’s a much simpler (and empirically, in my experience, more convincing) explanation of why we lose to even an AI with an IQ of 110. It is Dath Ilan, and we are Earth. Coordination is difficult for humans and the easy part for AIs.
I will note that Eliezer wrote That Alien Message a long time ago I think in part to try to convey the issue to this perspective, but it’s mostly about “information-theoretic bounds are probably not going to be tight” in a simulation-y universe instead of “here’s what coordination between computers looks like today”. I do predict the coordination point would be good to include in more of the intro materials.
This is 100% correct, and part of why I expect the focus on superintelligence, while literally true, is bad for AI outreach. There’s a much simpler (and empirically, in my experience, more convincing) explanation of why we lose to even an AI with an IQ of 110. It is Dath Ilan, and we are Earth. Coordination is difficult for humans and the easy part for AIs.
I will note that Eliezer wrote That Alien Message a long time ago I think in part to try to convey the issue to this perspective, but it’s mostly about “information-theoretic bounds are probably not going to be tight” in a simulation-y universe instead of “here’s what coordination between computers looks like today”. I do predict the coordination point would be good to include in more of the intro materials.