If I’m understanding the overall gist of this correctly (I’ve done a somewhat thorough skim), It is as follows:
Vitalik is (for the purpose of this essay) granting everything in the AI timeline up until doom. IE, Vitalik doesn’t necessarily agree with everything else, but sees a critical and under-appreciated weakness in the final steps where the misaligned superintelligent AI actually kills most-or-all humans.
This argument is developed by critiquing each tool the superintelligent AI has to do the job with (biotech, hacking, persuasion) & concluding that it is “far from a slam dunk”.
This seems wrong from where I’m standing. If misaligned superintelligence arises, the specific mechanism by which it chooses to kill humans is probably better than any specific plan we can come up with.
Furthermore, Vitalik’s counterarguments generally rely on defensive technology. What it doesn’t account for is that, in this scenario, all these defensive technologies would be coming from AI, and all the best AIs are allied in a coalition. If any one of these defensive technologies were a crucial blocker for the AI takeover, the AIs could fail to produce them, or produce poor versions.
For Vitalik’s picture to make sense, I think we need a much more multipolar future than what AI 2027 projects. It isn’t clear if we can achieve this, because even if there were hundreds or thousands of top AI companies rather than 3-5, they’d all be training AI with similar methods and similar data. We’ve already seen that data contamination can make one AI act like another AI.
If I’m understanding the overall gist of this correctly (I’ve done a somewhat thorough skim), It is as follows:
Vitalik is (for the purpose of this essay) granting everything in the AI timeline up until doom. IE, Vitalik doesn’t necessarily agree with everything else, but sees a critical and under-appreciated weakness in the final steps where the misaligned superintelligent AI actually kills most-or-all humans.
This argument is developed by critiquing each tool the superintelligent AI has to do the job with (biotech, hacking, persuasion) & concluding that it is “far from a slam dunk”.
This seems wrong from where I’m standing. If misaligned superintelligence arises, the specific mechanism by which it chooses to kill humans is probably better than any specific plan we can come up with.
Furthermore, Vitalik’s counterarguments generally rely on defensive technology. What it doesn’t account for is that, in this scenario, all these defensive technologies would be coming from AI, and all the best AIs are allied in a coalition. If any one of these defensive technologies were a crucial blocker for the AI takeover, the AIs could fail to produce them, or produce poor versions.
For Vitalik’s picture to make sense, I think we need a much more multipolar future than what AI 2027 projects. It isn’t clear if we can achieve this, because even if there were hundreds or thousands of top AI companies rather than 3-5, they’d all be training AI with similar methods and similar data. We’ve already seen that data contamination can make one AI act like another AI.