Depends a lot on what we mean by “smart” and “benevolent.”
For example, suppose by “smart” we mean not only intelligent, but intelligent enough to self-improve our intelligence in a for-practical-purposes-unbounded fashion. That seems at least roughly consistent with what we mean here by risky AI. And by “self-improve” here, we should include things like an agent building a system external to it, not just an agent modifying itself; that is, a system prohibited from editing its own source code but able to build another, smarter system and execute it should still count as self-improving for purposes of risk assessment.
And suppose that by “benevolent” we mean capable of consistently and reliably acting in the long-term best interests of humans.
Now, either (a) it’s possible for humans to build a smart machine, or (b) it’s not possible.
If (a), then humans are themselves a smart machine, so evolution is capable of building one. Whereas humans aren’t benevolent, despite what would seem to be significant selection pressure for benevolence operating over the same time-period. That suggests that benevolence is a harder problem.
If (b), then this argument doesn’t apply, and benevolence might be a harder or easier problem, or neither. That said, if (b), then it’s not clear why we ought to worry about AI risk at all.
Humans are semi-benevolent. I believe that if people in general didn’t do more good (behavior which leads towards human survival) than harm, the human race could not have existed as long as it has.
By observation, it’s not a matter of a small minority of people who make things a lot better vs. a majority whose effect is neutral or negative. I’m reasonably sure most people do more good than harm. The good that people do for themselves is included in the calculation.
This doesn’t mean people come anywhere near a theoretical maximum of benevolence. It just means that common behavior which doesn’t cause a problem doesn’t even get noticed.
I don’t know whether realizing this gives some way of applying leverage to get more benevolence, though I’m inclined to think that “build on what you’re doing well” is at least as good as “look at how awful you are”. (For the latter, consider the number of people who believe that if aliens met us, they’d destroy us out of disgust at how we treat each other.)
As I said initially, a lot depends on what we mean by “benevolent.” If we mean reliably doing more good for humans than harm, on average, then I agree that humans are benevolent (or “semibenevolent,” if you prefer) and suspect that building a benevolent (or semibenevolent) AGI is about as hard as building a smart one.
I agree that having a positive view of human nature has advantages over an equally accurate negative view.
And suppose that by “benevolent” we mean capable of consistently and reliably acting in the long-term best interests of humans.
...I in fact meant humans as individuals. And traits that act in the long-term best interests of individuals do in fact exert selection pressure on the genome.
But perhaps you’re suggesting that by “benevolent” I ought to have meant capable of consistently and reliably acting in the long-term best interests of humanity as a species, and not necessarily the individual?
Ah, I was thrown by the plural at the end of your definition.
But by saying humans aren’t benevolent, you mean that there are no/few humans for which it’s true that “this person consistently acts in hir own best interests?”
Yes, that’s what I mean. Actually, more broadly, I mean that there are no humans for which it’s true that they consistently act in anyone’s best interests, including themselves.
Depends a lot on what we mean by “smart” and “benevolent.”
For example, suppose by “smart” we mean not only intelligent, but intelligent enough to self-improve our intelligence in a for-practical-purposes-unbounded fashion. That seems at least roughly consistent with what we mean here by risky AI. And by “self-improve” here, we should include things like an agent building a system external to it, not just an agent modifying itself; that is, a system prohibited from editing its own source code but able to build another, smarter system and execute it should still count as self-improving for purposes of risk assessment.
And suppose that by “benevolent” we mean capable of consistently and reliably acting in the long-term best interests of humans.
Now, either (a) it’s possible for humans to build a smart machine, or (b) it’s not possible.
If (a), then humans are themselves a smart machine, so evolution is capable of building one. Whereas humans aren’t benevolent, despite what would seem to be significant selection pressure for benevolence operating over the same time-period. That suggests that benevolence is a harder problem.
If (b), then this argument doesn’t apply, and benevolence might be a harder or easier problem, or neither. That said, if (b), then it’s not clear why we ought to worry about AI risk at all.
Humans are semi-benevolent. I believe that if people in general didn’t do more good (behavior which leads towards human survival) than harm, the human race could not have existed as long as it has.
By observation, it’s not a matter of a small minority of people who make things a lot better vs. a majority whose effect is neutral or negative. I’m reasonably sure most people do more good than harm. The good that people do for themselves is included in the calculation.
This doesn’t mean people come anywhere near a theoretical maximum of benevolence. It just means that common behavior which doesn’t cause a problem doesn’t even get noticed.
I don’t know whether realizing this gives some way of applying leverage to get more benevolence, though I’m inclined to think that “build on what you’re doing well” is at least as good as “look at how awful you are”. (For the latter, consider the number of people who believe that if aliens met us, they’d destroy us out of disgust at how we treat each other.)
As I said initially, a lot depends on what we mean by “benevolent.” If we mean reliably doing more good for humans than harm, on average, then I agree that humans are benevolent (or “semibenevolent,” if you prefer) and suspect that building a benevolent (or semibenevolent) AGI is about as hard as building a smart one.
I agree that having a positive view of human nature has advantages over an equally accurate negative view.
“Whereas humans aren’t benevolent, despite what would seem to be significant selection pressure for benevolence operating over the same time-period.”
Evolution doesn’t act for the good of the species, so this looks wrong.
That’s an interesting distinction. When I said:
...I in fact meant humans as individuals. And traits that act in the long-term best interests of individuals do in fact exert selection pressure on the genome.
But perhaps you’re suggesting that by “benevolent” I ought to have meant capable of consistently and reliably acting in the long-term best interests of humanity as a species, and not necessarily the individual?
Ah, I was thrown by the plural at the end of your definition.
But by saying humans aren’t benevolent, you mean that there are no/few humans for which it’s true that “this person consistently acts in hir own best interests?”
Yes, that’s what I mean. Actually, more broadly, I mean that there are no humans for which it’s true that they consistently act in anyone’s best interests, including themselves.