Does Eliezer really not have other reasons beyond this epistemological view that he ought to update to ~99% based on his own inability to find a potentially-promising solution to the alignment problem over the course of his career?
I don’t really understand what kinds of reasons you think would justify having 99% confidence in an outcome. 99% is not very high confidence, in log-odds—I am much more than 99% confident in many claims. But, that aside, he has written millions of words on the subject, explaining his views in detail, including describing much of the enormous amount of evidence that he believes bears on this question. It is difficult to compress that evidence into a short summary. (Though there have been numerous attempts.)
And also because e.g. saying you’re 99% confident that building ASIs with the wrong goals would lead to human extinction because “It would be lethally dangerous to build ASIs that have the wrong goals” is circular and doesn’t actually explain why you’re so confident.
I mean, yes, I was trying to demonstrate that a short summary will obviously fail to convey information that most readers would find necessary to carry the argument (and that most readers would want different additional pieces of information from each other). However, “It would be lethally dangerous to build ASIs that have the wrong goals” is not circular. You might say it lacks justification, but many people have background beliefs such that a statement like that requires little or no additional justification[1].
99% is not very high confidence, in log-odds—I am much more than 99% confident in many claims.
I am too. But for how many of those beliefs that you’re 99+% sure of can you name several people like Paul Christiano who think you’re on the wrong-side-of-maybe about? For me, not a single example comes to mind.
However, “It would be lethally dangerous to build ASIs that have the wrong goals” is not circular. You might say it lacks justification
I agree that’s not circular. I meant that the full claim “building ASIs with the wrong goals would lead to human extinction because ‘It would be lethally dangerous to build ASIs that have the wrong goals’” is circular. “Lacks justification” would have been clearer.
For example, if they believe both that Drexlerian nanotechnology is possible and that the ASI in question would be able to build it.
I hold this background belief but don’t think that it means the original claim requires little additional justification. But getting into such details is beyond the scope of this discussion thread. (Brief gesture at an explanation: Even though humans could exterminate all the ants in a backyard when they build a house, they don’t. It similarly seems plausible to me that ASI could start building its factories on Earth to enable it to build von Neuman probes to begin colonizing the universe all without killing all humans on Earth. Maybe it’d extinct humanity by boiling the oceans like mentioned in IABIED, but I have enough doubt in these sorts of predictions to remain <<99% confident in the ‘It would be lethally dangerous [i.e. it’d lead to extinction] to build ASIs that have the wrong goals’ claim.)
Even though humans could exterminate all the ants in a backyard when they build a house, they don’t.
They do, however, exterminate all the ants (and many other species) in a lot more than a backyard when they use pesticides on a farm. Or members of a lot more species in an even wider area when they build a hydroelectric dam.
True, and humans do cause the extinction of some species globally too, not just in certain farm fields. But notably most species humans don’t cause the extinction of, so using the analogy with humans-animals as a reason to expect ASI to be 99% likely to extinct humanity doesn’t work. The analogy is merely suggestive of risk.
I don’t really understand what kinds of reasons you think would justify having 99% confidence in an outcome. 99% is not very high confidence, in log-odds—I am much more than 99% confident in many claims. But, that aside, he has written millions of words on the subject, explaining his views in detail, including describing much of the enormous amount of evidence that he believes bears on this question. It is difficult to compress that evidence into a short summary. (Though there have been numerous attempts.)
I mean, yes, I was trying to demonstrate that a short summary will obviously fail to convey information that most readers would find necessary to carry the argument (and that most readers would want different additional pieces of information from each other). However, “It would be lethally dangerous to build ASIs that have the wrong goals” is not circular. You might say it lacks justification, but many people have background beliefs such that a statement like that requires little or no additional justification[1].
For example, if they believe both that Drexlerian nanotechnology is possible and that the ASI in question would be able to build it.
Thanks for the replies.
I am too. But for how many of those beliefs that you’re 99+% sure of can you name several people like Paul Christiano who think you’re on the wrong-side-of-maybe about? For me, not a single example comes to mind.
I agree that’s not circular. I meant that the full claim “building ASIs with the wrong goals would lead to human extinction because ‘It would be lethally dangerous to build ASIs that have the wrong goals’” is circular. “Lacks justification” would have been clearer.
I hold this background belief but don’t think that it means the original claim requires little additional justification. But getting into such details is beyond the scope of this discussion thread. (Brief gesture at an explanation: Even though humans could exterminate all the ants in a backyard when they build a house, they don’t. It similarly seems plausible to me that ASI could start building its factories on Earth to enable it to build von Neuman probes to begin colonizing the universe all without killing all humans on Earth. Maybe it’d extinct humanity by boiling the oceans like mentioned in IABIED, but I have enough doubt in these sorts of predictions to remain <<99% confident in the ‘It would be lethally dangerous [i.e. it’d lead to extinction] to build ASIs that have the wrong goals’ claim.)
They do, however, exterminate all the ants (and many other species) in a lot more than a backyard when they use pesticides on a farm. Or members of a lot more species in an even wider area when they build a hydroelectric dam.
True, and humans do cause the extinction of some species globally too, not just in certain farm fields. But notably most species humans don’t cause the extinction of, so using the analogy with humans-animals as a reason to expect ASI to be 99% likely to extinct humanity doesn’t work. The analogy is merely suggestive of risk.