I’ve read the slides of the underlying talk, but not listened to it. I currently don’t expect to write a long response to this. My thoughts about points the talk touches on:
Existential risk vs. catastrophic risk. Often, there’s some question about whether or not existential risks are even possible. On slide 7 and 8 Sussman identifies a lot of reasons to think that humans cause catastrophic risks (ecological destruction could possibly kill 90% of people, but seems much more difficult for it to kill 100% of people), and the distinction between the two is only important if you think about the cosmic endowment. But of course if we think AI is an existential threat, and we think humans make AI, then it is true that humans present an existential threat to ourselves. I also note here that Sussman identifies synthetic biology as possibly an existential risk, which raises the question of why an AI couldn’t be a source of the existential risk presented by synthetic biology. (If an AI is built that wants to kill us, and that weapon is lying around, then we should be more concerned about AI because it has an opportunity.)
Accident risk vs. misuse risk. This article talks about it some, but the basic question is “will advanced AI cause problems because it did something no one wanted (accidents), or something bad people wanted (misuse)?”. Most technical AI safety research is focused on accident risk, for reasons that are too long to describe here, but it’s not crazy to be concerned about misuse risk, which seems to be Sussman’s primary focus. I also think the sort of accident risks that we’re concerned about require much deeper solutions that the normal sorts of bugs or accidents that one might imagine on hearing about this; the autonomous vehicle accident that occupies much of the talk is not a good testbed for thinking about what I think of as ‘accident risk’ and instead one should focus on something like the ‘nearest unblocked strategy’ article and related things.
Openness vs. closure. Open software allows for verifiability; I can know that lots of people have evaluated the decision-making of my self-driving car, rather than just Tesla’s internal programming team. But also open software allows for copying and modification; the software used to enable drones that deliver packages could be repurposed to enable drones that deliver hand grenades. If we think a technology is ‘dual use’, in that it can both be used to make things better (like printing DNA for medical treatments) and worse (like printing DNA to create new viruses), we generally don’t want those technologies to be open, and instead have carefully monitored access to dissuade improper use.
Solving near-term problems vs. long-term problems. Many people working on technical AI safety focus on applications with immediate uses, like the underlying math for how autonomous vehicles might play nicely with human drivers, and many people working on technical AI safety focus on research that will need to be done before we can safely deploy advanced artificial intelligence. Both of these problems seem real to me, and I wouldn’t dissuade someone from working on near-term safety work (especially if the alternative is that they do capabilities work!). I think that the ‘long-term’ here is measured in “low numbers of decades” instead of “low numbers of centuries,” and so it might be a mistake to call it ‘long-term,’ but the question of how to do prioritization here is actually somewhat complicated, and it seems better if we end up in a world where people working on near-term and long-term issues see each other as collaborators and allies instead of competitors for a limited supply of resources or attention.
I’ve read the slides of the underlying talk, but not listened to it. I currently don’t expect to write a long response to this. My thoughts about points the talk touches on:
Existential risk vs. catastrophic risk. Often, there’s some question about whether or not existential risks are even possible. On slide 7 and 8 Sussman identifies a lot of reasons to think that humans cause catastrophic risks (ecological destruction could possibly kill 90% of people, but seems much more difficult for it to kill 100% of people), and the distinction between the two is only important if you think about the cosmic endowment. But of course if we think AI is an existential threat, and we think humans make AI, then it is true that humans present an existential threat to ourselves. I also note here that Sussman identifies synthetic biology as possibly an existential risk, which raises the question of why an AI couldn’t be a source of the existential risk presented by synthetic biology. (If an AI is built that wants to kill us, and that weapon is lying around, then we should be more concerned about AI because it has an opportunity.)
Accident risk vs. misuse risk. This article talks about it some, but the basic question is “will advanced AI cause problems because it did something no one wanted (accidents), or something bad people wanted (misuse)?”. Most technical AI safety research is focused on accident risk, for reasons that are too long to describe here, but it’s not crazy to be concerned about misuse risk, which seems to be Sussman’s primary focus. I also think the sort of accident risks that we’re concerned about require much deeper solutions that the normal sorts of bugs or accidents that one might imagine on hearing about this; the autonomous vehicle accident that occupies much of the talk is not a good testbed for thinking about what I think of as ‘accident risk’ and instead one should focus on something like the ‘nearest unblocked strategy’ article and related things.
Openness vs. closure. Open software allows for verifiability; I can know that lots of people have evaluated the decision-making of my self-driving car, rather than just Tesla’s internal programming team. But also open software allows for copying and modification; the software used to enable drones that deliver packages could be repurposed to enable drones that deliver hand grenades. If we think a technology is ‘dual use’, in that it can both be used to make things better (like printing DNA for medical treatments) and worse (like printing DNA to create new viruses), we generally don’t want those technologies to be open, and instead have carefully monitored access to dissuade improper use.
Solving near-term problems vs. long-term problems. Many people working on technical AI safety focus on applications with immediate uses, like the underlying math for how autonomous vehicles might play nicely with human drivers, and many people working on technical AI safety focus on research that will need to be done before we can safely deploy advanced artificial intelligence. Both of these problems seem real to me, and I wouldn’t dissuade someone from working on near-term safety work (especially if the alternative is that they do capabilities work!). I think that the ‘long-term’ here is measured in “low numbers of decades” instead of “low numbers of centuries,” and so it might be a mistake to call it ‘long-term,’ but the question of how to do prioritization here is actually somewhat complicated, and it seems better if we end up in a world where people working on near-term and long-term issues see each other as collaborators and allies instead of competitors for a limited supply of resources or attention.