It’s also quite possible that some will be sadistic. Once powerful AI is in the picture, it also unlocks cheap, convenient, easy-to-apply, extremely potent brain-computer-interfaces that can mentally enslave people. And that sort of thing snowballs, because the more loyal-unto-death servants you have, the easier it is to kidnap and convert new people.
Along with other tech potentially unlocking things like immortality, and you have a recipe for things going quite badly if some sadist gets enough power to ramp into even more…
I mean, plus the weird race dynamics of the AI itself. Will the few controllers of AI cooperate peacefully, or occasionally get into arguments and get jealous of each other’s power? Might they one day get into a fight or significant competition that causes them to aim for even stronger AI servants to best their opponent, and thus leads to them losing control? Or a wide variety of other ways humans may fail to remain consistently sensible over a long period. It seems to me pretty likely that even 1 out of 1000 of the AI Lords losing control could easily lead to their uncontrolled AI self-improving enough to escape and conquer all the AI Lords. Just doesn’t seem like a sane and stable configuration for humanity to aim for, insofar as we are able to aim for anything.
The attractor basin around ‘genuinely nice value-aligned AI’ seems a lot more promising to me than ‘obedient AI controlled by centralized human power’. MIRI & co make arguments about a ‘near miss’ on value alignment being catastrophic, but after years of thought and debate on the subject, I’ve come around to disagreeing with this point. A really smart, really powerful AI that is trying its best to help humanity and satisfying humanity’s extrapolated values as best it can seems likely to… approach the problem intelligently. Like, recognize the need for epistemic humility and for continued human progress....
It’s also quite possible that some will be sadistic. Once powerful AI is in the picture, it also unlocks cheap, convenient, easy-to-apply, extremely potent brain-computer-interfaces that can mentally enslave people. And that sort of thing snowballs, because the more loyal-unto-death servants you have, the easier it is to kidnap and convert new people. Along with other tech potentially unlocking things like immortality, and you have a recipe for things going quite badly if some sadist gets enough power to ramp into even more… I mean, plus the weird race dynamics of the AI itself. Will the few controllers of AI cooperate peacefully, or occasionally get into arguments and get jealous of each other’s power? Might they one day get into a fight or significant competition that causes them to aim for even stronger AI servants to best their opponent, and thus leads to them losing control? Or a wide variety of other ways humans may fail to remain consistently sensible over a long period. It seems to me pretty likely that even 1 out of 1000 of the AI Lords losing control could easily lead to their uncontrolled AI self-improving enough to escape and conquer all the AI Lords. Just doesn’t seem like a sane and stable configuration for humanity to aim for, insofar as we are able to aim for anything. The attractor basin around ‘genuinely nice value-aligned AI’ seems a lot more promising to me than ‘obedient AI controlled by centralized human power’. MIRI & co make arguments about a ‘near miss’ on value alignment being catastrophic, but after years of thought and debate on the subject, I’ve come around to disagreeing with this point. A really smart, really powerful AI that is trying its best to help humanity and satisfying humanity’s extrapolated values as best it can seems likely to… approach the problem intelligently. Like, recognize the need for epistemic humility and for continued human progress....