Given that the basic case for x-risks is so simple/obvious[1], I think most people arguing against any risk are probably doing so due to some kind of myopic/irrational subconscious motive.
It isn’t simple or obvious to many people. I’ve discussed it with an open-minded philosophy professor and he had many doubts, like:
doubts about the feasibility of building AGI or ASI (he had read objections like Searle’s Chinese Room and didn’t know what ChatGPT is capable of currently)
doubts about such an AI having goals
doubts about the plausibility of an ASI wanting us dead, due to his credence in moral realism
doubts about the feasibility of the AI gaining power (he asked “How would it get all the energy? Couldn’t we just unplug the data center or whatever?”)
doubts about this being more concerning than mainstream risks, like autonomous weapons
So far I’ve had answers to these things, but they required their own long discussions, and the thornier ones (like moral realism) didn’t get resolved. Overall, he seems to take it somewhat seriously, but he also has lots of experience with philosophers, students, coworkers, etc. trying to convince him of weird things, so it’s unfortunately understandable that he isn’t that concerned about this thing in particular yet.
I suppose you could argue that all of his objections are trivial and he’s obviously biased, but I don’t think that tackling his emotions instead of his arguments would help much.
Philosophers have come up with a bunch of elaborate, if flawed, arguments for moral realism over the years. This professor gave me the book The Moral Universe which is a recent instance of this. To be fair, people who haven’t already gotten got by modern philosophy or religion can be sold a form of anti-realism with simple thought experiments, like the aliens who desire nests with prime-numbered stones from IABIED.
I think moral realism is something many people believe for emotional reasons (“How DARE you suggest otherwise?”), but it’s also a conclusion that can be gotten to with subtly flawed abstract reasoning.
You could probably sidestep the moral realism debate when talking about x-risk, because it seems plausible that AI could be wrong about morality, or it could simply be an unfeeling force of nature to which moral reasoning doesn’t apply. I’m realizing now that if I wasn’t so eager to debate morality, I could’ve avoided it altogether.
It isn’t simple or obvious to many people. I’ve discussed it with an open-minded philosophy professor and he had many doubts, like:
doubts about the feasibility of building AGI or ASI (he had read objections like Searle’s Chinese Room and didn’t know what ChatGPT is capable of currently)
doubts about such an AI having goals
doubts about the plausibility of an ASI wanting us dead, due to his credence in moral realism
doubts about the feasibility of the AI gaining power (he asked “How would it get all the energy? Couldn’t we just unplug the data center or whatever?”)
doubts about this being more concerning than mainstream risks, like autonomous weapons
So far I’ve had answers to these things, but they required their own long discussions, and the thornier ones (like moral realism) didn’t get resolved. Overall, he seems to take it somewhat seriously, but he also has lots of experience with philosophers, students, coworkers, etc. trying to convince him of weird things, so it’s unfortunately understandable that he isn’t that concerned about this thing in particular yet.
I suppose you could argue that all of his objections are trivial and he’s obviously biased, but I don’t think that tackling his emotions instead of his arguments would help much.
I updated a bit towards thinking that incompetence-at-reasoning is a more common/influential factor than I previously thought. Thanks.
However: Where do you think that moral realism comes from? Why is it a “thorny” issue?
Philosophers have come up with a bunch of elaborate, if flawed, arguments for moral realism over the years. This professor gave me the book The Moral Universe which is a recent instance of this. To be fair, people who haven’t already gotten got by modern philosophy or religion can be sold a form of anti-realism with simple thought experiments, like the aliens who desire nests with prime-numbered stones from IABIED.
I think moral realism is something many people believe for emotional reasons (“How DARE you suggest otherwise?”), but it’s also a conclusion that can be gotten to with subtly flawed abstract reasoning.
You could probably sidestep the moral realism debate when talking about x-risk, because it seems plausible that AI could be wrong about morality, or it could simply be an unfeeling force of nature to which moral reasoning doesn’t apply. I’m realizing now that if I wasn’t so eager to debate morality, I could’ve avoided it altogether.