I should probably engage more deeply with the basin of good reference post. But I currently thank 5-10 years is not nearly enough that “we” can confirm that the AI is in the basin of good deference. Sure, we can maybe prove with high confidence that the AI is not actively scheming against us, and maybe even the voters will be justified in believing the narrow claim from the scientists that the AI is not actively scheming. But otherwise, I very much sympathize with Richard Ngo’s point that there is no way for most normal people to understand and consent to this process so quickly. 5-10 is like two election cycles! It’s really not very much time! And I’m not just talking about the random voters, I feel that I myself will be very distrustful of the Jupiter-brained creature we are creating for whom the concept of instruction following breaks down.
I don’t have great ideas what to do instead, but I think probably gradualism is the best approach. The one type of alignment that people have experience with is raising children. The one type of value evolution people feel relatively fine with is the new generation of people gradually shaping culture in their own way.
So let’s just have some kids first who can grow up in a more abundant society, built by the limited involvement of TED AIs. Then we have guided humanity through the times of peril, and solved our own small portion of the alignment problem by lovingly raising a generation of somewhat smarter and better-adjusted people, who can then carry on the torch. Solving the rest of the alignment problem is their job, not ours.
Tentatively, I think that after one or two generations, people should start using a moderate amount of human genetic engineering to shape the next generation to be even smarter, wiser and happier. Then at some point, probably one generation should start merging with the machines and becoming cyborgs. I don’t know what happens after that, it’s their choice. And throughout this process, I hope that there always remain some people (perhaps not very many, probably keeping only a small sliver of the stars to themselves), who wish not to progress to the next level of development, and wish to remain in their only more-or-less enhance human form, but who will still love the people who progress one step further in the next generation, and who will still be loved by them in turn.
It sounds like your high level plan is to make artificial intelligence smart enough to stabilize the situation and no smarter. Then do relatively chill human intelligence augmentation & cyborgism until eventually the augmented humans/cyborgs are ready to build superintelligence.
I don’t think this is a crazy plan. It might be better than Plan A. I guess my main worries would be
Exogneous xrisks that TED AIs can’t manage down to zero get us while we’re doing chill intelligence augmentation.
The limits of machine intelligence prove to be so far from the limits of augmented human intelligence that we’re never ready to bulid superintelligence, and maybe we miss out on some unique benefits that superintelligence would have provided.
I think exogneoue xrisk is probably very small, especially that I’m in favor of starting some chill levels od space colonization (e.g. Mars) pretty soon.
It’s not impossible but I would find it very surprising if your TED AI collective was unwise or untrustrworthy enough that it could drift into misalignment over time without stopping itself, but trustworthy and wise enough that we could have handed off the intelligrnt explosion to them, and they would have safely got us to the limits of intelligence.
Yep, the third point is a serious objection, but I feel that if our descendants never feel ready to buikd superintelligence, then maybe they just shouldn’t. I think it’s plausible that TED AI plus maybe narrow superintelligences (e.g. specialized in space-probe design) are enough for a near maximally good future.
I should probably engage more deeply with the basin of good reference post. But I currently thank 5-10 years is not nearly enough that “we” can confirm that the AI is in the basin of good deference. Sure, we can maybe prove with high confidence that the AI is not actively scheming against us, and maybe even the voters will be justified in believing the narrow claim from the scientists that the AI is not actively scheming. But otherwise, I very much sympathize with Richard Ngo’s point that there is no way for most normal people to understand and consent to this process so quickly. 5-10 is like two election cycles! It’s really not very much time! And I’m not just talking about the random voters, I feel that I myself will be very distrustful of the Jupiter-brained creature we are creating for whom the concept of instruction following breaks down.
I don’t have great ideas what to do instead, but I think probably gradualism is the best approach. The one type of alignment that people have experience with is raising children. The one type of value evolution people feel relatively fine with is the new generation of people gradually shaping culture in their own way.
So let’s just have some kids first who can grow up in a more abundant society, built by the limited involvement of TED AIs. Then we have guided humanity through the times of peril, and solved our own small portion of the alignment problem by lovingly raising a generation of somewhat smarter and better-adjusted people, who can then carry on the torch. Solving the rest of the alignment problem is their job, not ours.
Tentatively, I think that after one or two generations, people should start using a moderate amount of human genetic engineering to shape the next generation to be even smarter, wiser and happier. Then at some point, probably one generation should start merging with the machines and becoming cyborgs. I don’t know what happens after that, it’s their choice. And throughout this process, I hope that there always remain some people (perhaps not very many, probably keeping only a small sliver of the stars to themselves), who wish not to progress to the next level of development, and wish to remain in their only more-or-less enhance human form, but who will still love the people who progress one step further in the next generation, and who will still be loved by them in turn.
It sounds like your high level plan is to make artificial intelligence smart enough to stabilize the situation and no smarter. Then do relatively chill human intelligence augmentation & cyborgism until eventually the augmented humans/cyborgs are ready to build superintelligence.
I don’t think this is a crazy plan. It might be better than Plan A. I guess my main worries would be
Exogneous xrisks that TED AIs can’t manage down to zero get us while we’re doing chill intelligence augmentation.
The TED AIs somehow drift into misalignment.
The limits of machine intelligence prove to be so far from the limits of augmented human intelligence that we’re never ready to bulid superintelligence, and maybe we miss out on some unique benefits that superintelligence would have provided.
I think exogneoue xrisk is probably very small, especially that I’m in favor of starting some chill levels od space colonization (e.g. Mars) pretty soon.
It’s not impossible but I would find it very surprising if your TED AI collective was unwise or untrustrworthy enough that it could drift into misalignment over time without stopping itself, but trustworthy and wise enough that we could have handed off the intelligrnt explosion to them, and they would have safely got us to the limits of intelligence.
Yep, the third point is a serious objection, but I feel that if our descendants never feel ready to buikd superintelligence, then maybe they just shouldn’t. I think it’s plausible that TED AI plus maybe narrow superintelligences (e.g. specialized in space-probe design) are enough for a near maximally good future.