I vaguely agree, but also I think it’s not so implausible to get a pretty robust stop. I don’t have a specific way, but think for example how, for now, it’s hard to make a global pandemic from your garage, even though that’s probably feasible in principle. Some combination of, no one would do that, and if they would, no one would help them, and without help it would just be hard. I mean, that’s not very reassuring; largely I’m like “IDK, humanity (including HIA people) might figure it out!”.
I think it is far easier to mistakenly/motivatedly conclude that actually superintelligence is a grand thing to aim at, and that the risks are minimal etc., than it is for creating global pandemics. As such I think the latter requires people who are deluded and/or evil to a quite unusual degree, whereas the former strikes me as requiring only people deluded and/or evil to an exceedingly common degree. Which makes me more worried that social pressure will not suffice to prevent it.
I kinda agree, and in general I definitely agree it’s probably much harder than in the case of pandemics. Partly I think this is something we can change (by explaining risks and offering alternatives). On the other hand, as Critch points out, we do have quite strong and functionally operative intuitions around cutting down the tall poppies. In many cases this is a bad intuition, but it’s not 100% wholly without merit in general, and in this case it would fit well.
I vaguely agree, but also I think it’s not so implausible to get a pretty robust stop. I don’t have a specific way, but think for example how, for now, it’s hard to make a global pandemic from your garage, even though that’s probably feasible in principle. Some combination of, no one would do that, and if they would, no one would help them, and without help it would just be hard. I mean, that’s not very reassuring; largely I’m like “IDK, humanity (including HIA people) might figure it out!”.
I think it is far easier to mistakenly/motivatedly conclude that actually superintelligence is a grand thing to aim at, and that the risks are minimal etc., than it is for creating global pandemics. As such I think the latter requires people who are deluded and/or evil to a quite unusual degree, whereas the former strikes me as requiring only people deluded and/or evil to an exceedingly common degree. Which makes me more worried that social pressure will not suffice to prevent it.
I kinda agree, and in general I definitely agree it’s probably much harder than in the case of pandemics. Partly I think this is something we can change (by explaining risks and offering alternatives). On the other hand, as Critch points out, we do have quite strong and functionally operative intuitions around cutting down the tall poppies. In many cases this is a bad intuition, but it’s not 100% wholly without merit in general, and in this case it would fit well.