Yes, as I said, you seem to assume that it is very likely to succeed on all the hard problems but yet fail on the scope boundary. The scary idea states that it is likely that if we create self-improving AI it will consume humanity. I believe that is a rather unlikely outcome and haven’t seen any good reason to believe something else yet.
The scary idea states that it is likely that if we create self-improving AI it will consume humanity.
No, it states that we run the risk of accidentally making something that will consume (or exterminate, subvert, betray, make miserable, or otherwise Do Bad Things to) humanity, that looks perfectly safe and correct, right up until it’s too late to do anything about it… and that this is the default case: the case if we don’t do something extraordinary to prevent it.
This doesn’t require self-improvement, and it doesn’t require wiping out humanity. It just requires normal, every-day human error.
SIAI’s “Scary Idea”, which is the idea that: progressing toward advanced AGI without a design for “provably non-dangerous AGI” (or something closely analogous, often called “Friendly AI” in SIAI lingo) is highly likely to lead to an involuntary end for the human race.
Yes, as I said, you seem to assume that it is very likely to succeed on all the hard problems but yet fail on the scope boundary. The scary idea states that it is likely that if we create self-improving AI it will consume humanity. I believe that is a rather unlikely outcome and haven’t seen any good reason to believe something else yet.
No, it states that we run the risk of accidentally making something that will consume (or exterminate, subvert, betray, make miserable, or otherwise Do Bad Things to) humanity, that looks perfectly safe and correct, right up until it’s too late to do anything about it… and that this is the default case: the case if we don’t do something extraordinary to prevent it.
This doesn’t require self-improvement, and it doesn’t require wiping out humanity. It just requires normal, every-day human error.
Here is Ben’s phrasing: