I think the best counter-argument is that, as James Miller points out in a similar comment, there is almost certainly some positive luck in creating a Von Neumann (or a Christiano) and only a small reduction in ability takes somebody to merely one among thousands of peers. With enough clones, one should expect a few serious successes, but if we create lots of capable AI researchers for only a few people capable of making unusual leaps in safety then the tradeoff would not be very clear (though I would guess still positive given, as Matthew points out, heritability of most personality traits).
The better option, though more extreme, would be making people who we were confident would be above the normal range of human abilities. I think this would answer Yair’s worry that more smart people would be just as likely to accelerate unsafe as safe AI. If there were a small, connected group of people with far superior ability to create both safe and unsafe AI, the strength of competition would not be as strong and the amount of slack and ease of cooperation amongst those at the frontier would be greater. Also, while intelligence is not directly related to being right, or altruistic; it’s in everybody’s personal interest not to create unaligned AI, and we would expect these people to be good at identifying risks to themselves. Even if they were to attempt to use it for their own ends, this is likely to be better for humanity than AI not aligned to any human.
The more promising pathway towards this seems to be iterated embryo selection on polygenic scores for IQ (+ any other trait expected to be helpful) through in vitro gametogenesis, which is not currently possible but may be possible within a decade or so and could generate people with intelligence well beyond current human limits. Gwern has a gigantic write-up of the subject here if anyone is interested, which I won’t try to summarize.
I think this is worth taking seriously.
I think the best counter-argument is that, as James Miller points out in a similar comment, there is almost certainly some positive luck in creating a Von Neumann (or a Christiano) and only a small reduction in ability takes somebody to merely one among thousands of peers. With enough clones, one should expect a few serious successes, but if we create lots of capable AI researchers for only a few people capable of making unusual leaps in safety then the tradeoff would not be very clear (though I would guess still positive given, as Matthew points out, heritability of most personality traits).
The better option, though more extreme, would be making people who we were confident would be above the normal range of human abilities. I think this would answer Yair’s worry that more smart people would be just as likely to accelerate unsafe as safe AI. If there were a small, connected group of people with far superior ability to create both safe and unsafe AI, the strength of competition would not be as strong and the amount of slack and ease of cooperation amongst those at the frontier would be greater. Also, while intelligence is not directly related to being right, or altruistic; it’s in everybody’s personal interest not to create unaligned AI, and we would expect these people to be good at identifying risks to themselves. Even if they were to attempt to use it for their own ends, this is likely to be better for humanity than AI not aligned to any human.
The more promising pathway towards this seems to be iterated embryo selection on polygenic scores for IQ (+ any other trait expected to be helpful) through in vitro gametogenesis, which is not currently possible but may be possible within a decade or so and could generate people with intelligence well beyond current human limits. Gwern has a gigantic write-up of the subject here if anyone is interested, which I won’t try to summarize.