What I mean by “nice” is roughly the opposite of being a ruthless sociopath. It means treating other sentient beings well for its own sake.
Most humans are definitely not ruthless sociopaths. Sociopaths are estimated at about 10% of the population. And most of those aren’t even that ruthless; I think it’s a spectrum, like all biological mental differences. This leaves the conclusion that even NON-sociopathic humans are often pretty ruthless when they can get away with it, like when they hold a lot of power. But that’s pretty much beside the main point here, which is that we shouldn’t expect nice/non-ruthless behavior by default.
Laws and norms are not going to restrain an AGI that can route around them easily when it’s smart enough. They barely restrain human sociopaths who are bad at routing around them.
You might envision AGIs enforcing laws that include human well-being, and creating a nice just society roughly for the same reasons we have now. But if those AGIs don’t genuinely care about humans, I’d strongly expect humans to soon occupy the legal positions of farm animals—or perhaps worse, the many species we’ve extincted even though we like them, because they don’t provide our society any real benefit, and we wanted to do other stuff with their habitats.
That’s why we’re kind of obsessed with aligning smarter than human AI so it genuinely and intrinsically cares about us, or at least reliably takes our orders as intended.
What I mean by “nice” is roughly the opposite of being a ruthless sociopath. It means treating other sentient beings well for its own sake.
Most humans are definitely not ruthless sociopaths. Sociopaths are estimated at about 10% of the population. And most of those aren’t even that ruthless; I think it’s a spectrum, like all biological mental differences. This leaves the conclusion that even NON-sociopathic humans are often pretty ruthless when they can get away with it, like when they hold a lot of power. But that’s pretty much beside the main point here, which is that we shouldn’t expect nice/non-ruthless behavior by default.
Laws and norms are not going to restrain an AGI that can route around them easily when it’s smart enough. They barely restrain human sociopaths who are bad at routing around them.
You might envision AGIs enforcing laws that include human well-being, and creating a nice just society roughly for the same reasons we have now. But if those AGIs don’t genuinely care about humans, I’d strongly expect humans to soon occupy the legal positions of farm animals—or perhaps worse, the many species we’ve extincted even though we like them, because they don’t provide our society any real benefit, and we wanted to do other stuff with their habitats.
That’s why we’re kind of obsessed with aligning smarter than human AI so it genuinely and intrinsically cares about us, or at least reliably takes our orders as intended.