The claim isn’t that minds are safe and nice by default. It’s that they’re not sociopaths.
If in your view, most humans are basically ruthless sociopaths, then that’s good news, isn’t it? Sociopathic AIs would fit in well to our culture. It would mean our laws and norms do a remarkably good job of restraining us, so there’d be hope they’d do the same for future AIs.
What I mean by “nice” is roughly the opposite of being a ruthless sociopath. It means treating other sentient beings well for its own sake.
Most humans are definitely not ruthless sociopaths. Sociopaths are estimated at about 10% of the population. And most of those aren’t even that ruthless; I think it’s a spectrum, like all biological mental differences. This leaves the conclusion that even NON-sociopathic humans are often pretty ruthless when they can get away with it, like when they hold a lot of power. But that’s pretty much beside the main point here, which is that we shouldn’t expect nice/non-ruthless behavior by default.
Laws and norms are not going to restrain an AGI that can route around them easily when it’s smart enough. They barely restrain human sociopaths who are bad at routing around them.
You might envision AGIs enforcing laws that include human well-being, and creating a nice just society roughly for the same reasons we have now. But if those AGIs don’t genuinely care about humans, I’d strongly expect humans to soon occupy the legal positions of farm animals—or perhaps worse, the many species we’ve extincted even though we like them, because they don’t provide our society any real benefit, and we wanted to do other stuff with their habitats.
That’s why we’re kind of obsessed with aligning smarter than human AI so it genuinely and intrinsically cares about us, or at least reliably takes our orders as intended.
The claim isn’t that minds are safe and nice by default. It’s that they’re not sociopaths.
I thought one of the tenets of this debate is that there’s no in-between. Either safe and nice (aligned) or everybody dies (not aligned). Humans are a good example—most are not pure psychopaths, and yet they do a ton of harm to each other all the time, and have threatened to destroy the species for decades. A set of much more powerful minds with even that level of misalignment would be disaster, and if they’re slightly worse than humans, so much the worse.
The claim isn’t that minds are safe and nice by default. It’s that they’re not sociopaths.
If in your view, most humans are basically ruthless sociopaths, then that’s good news, isn’t it? Sociopathic AIs would fit in well to our culture. It would mean our laws and norms do a remarkably good job of restraining us, so there’d be hope they’d do the same for future AIs.
What I mean by “nice” is roughly the opposite of being a ruthless sociopath. It means treating other sentient beings well for its own sake.
Most humans are definitely not ruthless sociopaths. Sociopaths are estimated at about 10% of the population. And most of those aren’t even that ruthless; I think it’s a spectrum, like all biological mental differences. This leaves the conclusion that even NON-sociopathic humans are often pretty ruthless when they can get away with it, like when they hold a lot of power. But that’s pretty much beside the main point here, which is that we shouldn’t expect nice/non-ruthless behavior by default.
Laws and norms are not going to restrain an AGI that can route around them easily when it’s smart enough. They barely restrain human sociopaths who are bad at routing around them.
You might envision AGIs enforcing laws that include human well-being, and creating a nice just society roughly for the same reasons we have now. But if those AGIs don’t genuinely care about humans, I’d strongly expect humans to soon occupy the legal positions of farm animals—or perhaps worse, the many species we’ve extincted even though we like them, because they don’t provide our society any real benefit, and we wanted to do other stuff with their habitats.
That’s why we’re kind of obsessed with aligning smarter than human AI so it genuinely and intrinsically cares about us, or at least reliably takes our orders as intended.
I thought one of the tenets of this debate is that there’s no in-between. Either safe and nice (aligned) or everybody dies (not aligned). Humans are a good example—most are not pure psychopaths, and yet they do a ton of harm to each other all the time, and have threatened to destroy the species for decades. A set of much more powerful minds with even that level of misalignment would be disaster, and if they’re slightly worse than humans, so much the worse.