and find other robust ways to prevent the creation of AGI
Well, if there are robust ways to prevent the creation of AGI, that’s a crux for me. But it seems pretty hard to do if it is possible in principle to train superintelligence on a laptop.
If you’re going to safe the world by making aligned AGI, that works by then doing something with the AGI; whatever that action is, humans could also do it (though it could be much harder).
Social pressure and social avoidance can be quite powerful. Like, imagine that every person knows for sure that if anyone knew that ze is thinking about how to make AGI, they would hate / shun zer. So now it’s logistically very difficult / infeasible to talk with other people about your AGI ideas. That makes it way way way harder to make long term progress on the technical problem of making AGI.
The motive to make AGI can be sapped, perhaps to a very large degree, by clarifying how dangerous it is and by offering hopeworthy alternatives.
You could have technical surveillance, a la Critch (IIUC) as presented in his interview with Shapira. Sounds hard but could be feasible.
I think it is conceivable that humanity could learn to somehow screen enough of the computations occuring on each laptop, enough of the thoughts occuring in each brain, etc., for danger in a way that is not obviously more dystopian/rights-violating than the IAEA. I do not however currently consider it obvious that this is possible, nor that humanity will try to (and succeed at) figuring out how in time, and on my models this is roughly the degree of screening that seems likely necessary to prevent the creation of superintelligence for centuries, rather than years or low-decades.
Well I agree it’s not super obvious? I mean I’m not claiming that I have a bunch of inferences that you don’t, or that I’m confident it’s doable. But like, society can make an intellectual project way way way harder just by being hostile toward it.
I vaguely agree, but also I think it’s not so implausible to get a pretty robust stop. I don’t have a specific way, but think for example how, for now, it’s hard to make a global pandemic from your garage, even though that’s probably feasible in principle. Some combination of, no one would do that, and if they would, no one would help them, and without help it would just be hard. I mean, that’s not very reassuring; largely I’m like “IDK, humanity (including HIA people) might figure it out!”.
I think it is far easier to mistakenly/motivatedly conclude that actually superintelligence is a grand thing to aim at, and that the risks are minimal etc., than it is for creating global pandemics. As such I think the latter requires people who are deluded and/or evil to a quite unusual degree, whereas the former strikes me as requiring only people deluded and/or evil to an exceedingly common degree. Which makes me more worried that social pressure will not suffice to prevent it.
I kinda agree, and in general I definitely agree it’s probably much harder than in the case of pandemics. Partly I think this is something we can change (by explaining risks and offering alternatives). On the other hand, as Critch points out, we do have quite strong and functionally operative intuitions around cutting down the tall poppies. In many cases this is a bad intuition, but it’s not 100% wholly without merit in general, and in this case it would fit well.
Well, if there are robust ways to prevent the creation of AGI, that’s a crux for me. But it seems pretty hard to do if it is possible in principle to train superintelligence on a laptop.
To expand a bit more:
If you’re going to safe the world by making aligned AGI, that works by then doing something with the AGI; whatever that action is, humans could also do it (though it could be much harder).
Social pressure and social avoidance can be quite powerful. Like, imagine that every person knows for sure that if anyone knew that ze is thinking about how to make AGI, they would hate / shun zer. So now it’s logistically very difficult / infeasible to talk with other people about your AGI ideas. That makes it way way way harder to make long term progress on the technical problem of making AGI.
The motive to make AGI can be sapped, perhaps to a very large degree, by clarifying how dangerous it is and by offering hopeworthy alternatives.
You could have technical surveillance, a la Critch (IIUC) as presented in his interview with Shapira. Sounds hard but could be feasible.
I think it is conceivable that humanity could learn to somehow screen enough of the computations occuring on each laptop, enough of the thoughts occuring in each brain, etc., for danger in a way that is not obviously more dystopian/rights-violating than the IAEA. I do not however currently consider it obvious that this is possible, nor that humanity will try to (and succeed at) figuring out how in time, and on my models this is roughly the degree of screening that seems likely necessary to prevent the creation of superintelligence for centuries, rather than years or low-decades.
Well I agree it’s not super obvious? I mean I’m not claiming that I have a bunch of inferences that you don’t, or that I’m confident it’s doable. But like, society can make an intellectual project way way way harder just by being hostile toward it.
I vaguely agree, but also I think it’s not so implausible to get a pretty robust stop. I don’t have a specific way, but think for example how, for now, it’s hard to make a global pandemic from your garage, even though that’s probably feasible in principle. Some combination of, no one would do that, and if they would, no one would help them, and without help it would just be hard. I mean, that’s not very reassuring; largely I’m like “IDK, humanity (including HIA people) might figure it out!”.
I think it is far easier to mistakenly/motivatedly conclude that actually superintelligence is a grand thing to aim at, and that the risks are minimal etc., than it is for creating global pandemics. As such I think the latter requires people who are deluded and/or evil to a quite unusual degree, whereas the former strikes me as requiring only people deluded and/or evil to an exceedingly common degree. Which makes me more worried that social pressure will not suffice to prevent it.
I kinda agree, and in general I definitely agree it’s probably much harder than in the case of pandemics. Partly I think this is something we can change (by explaining risks and offering alternatives). On the other hand, as Critch points out, we do have quite strong and functionally operative intuitions around cutting down the tall poppies. In many cases this is a bad intuition, but it’s not 100% wholly without merit in general, and in this case it would fit well.