I don’t think we need to explicitly plan for the later stages. If we have a sufficiently advanced AI that we know is aligned and capable of intelligently self-modifying without becoming unaligned, we can probably put more confidence in the seed AI’s ability to construct the final stages than in our ability to shape the seed AI to better construct the final stages.
Edit: that’s insufficient. What I mean is that once you make the seed AI I described, any change you make to the seed AI that’s explicitly for the purpose of guiding its takeoff will be practically useless and possibly harmful given the AI’s advantage. I think we may reach a point where we can trust the seed AI to do the job well better than we can trust ourselves to do the job well.
I don’t think we need to explicitly plan for the later stages. If we have a sufficiently advanced AI that we know is aligned and capable of intelligently self-modifying without becoming unaligned, we can probably put more confidence in the seed AI’s ability to construct the final stages than in our ability to shape the seed AI to better construct the final stages.
Edit: that’s insufficient. What I mean is that once you make the seed AI I described, any change you make to the seed AI that’s explicitly for the purpose of guiding its takeoff will be practically useless and possibly harmful given the AI’s advantage. I think we may reach a point where we can trust the seed AI to do the job well better than we can trust ourselves to do the job well.