Scientific-style risk assessments should be positioned in the frame of some science. For all the examples that you gave (atomic bomb, vacuum decay, etc.) that was obviously physics...No such general science of intelligence exists at the moment.
This is a very good point. IDK how much less true this is becoming, but I agree that one reason it’s hard to make a comprehensive quantitative caser rather than a scattered (though related) set of less rigorous arguments is that there are so many unknown interactions. The physicists inventing atomic weapons knew they needed to be sure they wouldn’t ignite the atmosphere, but at least they knew what it meant to calculate that. Biologists and epidemiologists may not be able to immediately tell how dangerous a new pathogen will be, but at least they have a lot of data and research on existing pathogens and past outbreaks to draw from.
Instead, we’re left relying on more abstract forms of reasoning, which feel less convincing to most people and are much more open to disagreement about reference classes and background assumptions.
Also, AFAICT, TC’s base rate for AI being dangerous seems to be something analogous to, “Well, past technologies have been good on net, even dangerous ones, so [by some intuitive analog of Laplace’s rule of succession] we should expect this to turn out fine.” Whereas mine is more like, “Well, evolution of new hominins has never gone well for past hominins (or great apes) so why should this new increase in intelligence go better for us?” combined with, “Well, we’ve never yet been able to write software that does exactly what we want it to for anything complex, or been able to prevent other humans from misusing and breaking it, so why should this be different?”
Also, AFAICT, TC’s base rate for AI being dangerous seems to be something analogous to, “Well, past technologies have been good on net, even dangerous ones, so [by some intuitive analog of Laplace’s rule of succession] we should expect this to turn out fine.” Whereas mine is more like, “Well, evolution of new hominins has never gone well for past hominins (or great apes) so why should this new increase in intelligence go better for us?” combined with, “Well, we’ve never yet been able to write software that does exactly what we want it to for anything complex, or been able to prevent other humans from misusing and breaking it, so why should this be different?”
Tyler Cohen. I only used initials because the OP did the same.
And yes, I read that post, and I’ve seen similar arguments a number of times, and not just recently. They’re getting a lot sharper recently for obvious reasons, though.
I don’t think the base rates are crazy—the new evolution of hominins one is only wrong if you forget who ‘you’ is. TC and many other people are assuming that ‘we’ will be the ‘you’ that are evolving. (The worry among people here is that ‘they’ will have their own ‘you’.)
And the second example, writing new software that breaks—that is the same as making any new technology, we have done this before, and we were fine last time. Yes there were computer viruses, yes some people lost fingers in looms back in the day. But it was okay in the long run.
I think people arguing against these base rates need to do more work. The base rates are reasonable, it is the lack of updating that makes the difference. So lets help them update!
I think updating against these base rates is the critical thing.
But it’s not really an update. The key difference between optimists and pessimists in this area is the recognition that there are no base rates for something like AGI. We have developed new technologies before, but we have never developed a new species before.
New forms of intelligence and agency are a completely new phenomena. Sonic you wanted to ascribe a base rate of our surviving this with zero previous examples, you’d put it at .5. if you counted all of the previous hominid extinctions as relevant, you’d actually put the base rate much lower.
This really seems like the relevant comparison. Tools don’t kill you, but strange creatures do. AGI will be a creature, not a tool.
Instead, we’re left relying on more abstract forms of reasoning
See, the frustrating thing is, I really don’t think we are! There are loads of clear, concrete things that can be picked out and expanded upon. (See my sibling comment also.)
Honestly not sure if I agree or not, but even if true, it’s very hard to convince most people even with lots of real-world examples and data. Just ask anyone with an interest in the comparative quantitative risk assessment of different electricity sources, or ways of handling waste, and then ask them about the process of getting that permitted and built. And really, could you imagine if we subjected AI labs even to just 10% of the regulation we put in the way of letting people add a bedroom or bathroom to their houses?
Also, it doesn’t take a whole lot of abstraction to be more abstract than the physics examples I was responding too, and even then I don’t think we had nearly as much concrete data as we probably should have about the atmospheric nitrogen question. (Note that the H bomb developers also did the math that made them think Lithium-6 won’t contribute to yield, and were wrong. Not nearly as high stakes, so maybe they weren’t as careful? But still disconcerting).
This is a very good point. IDK how much less true this is becoming, but I agree that one reason it’s hard to make a comprehensive quantitative caser rather than a scattered (though related) set of less rigorous arguments is that there are so many unknown interactions. The physicists inventing atomic weapons knew they needed to be sure they wouldn’t ignite the atmosphere, but at least they knew what it meant to calculate that. Biologists and epidemiologists may not be able to immediately tell how dangerous a new pathogen will be, but at least they have a lot of data and research on existing pathogens and past outbreaks to draw from.
Instead, we’re left relying on more abstract forms of reasoning, which feel less convincing to most people and are much more open to disagreement about reference classes and background assumptions.
Also, AFAICT, TC’s base rate for AI being dangerous seems to be something analogous to, “Well, past technologies have been good on net, even dangerous ones, so [by some intuitive analog of Laplace’s rule of succession] we should expect this to turn out fine.” Whereas mine is more like, “Well, evolution of new hominins has never gone well for past hominins (or great apes) so why should this new increase in intelligence go better for us?” combined with, “Well, we’ve never yet been able to write software that does exactly what we want it to for anything complex, or been able to prevent other humans from misusing and breaking it, so why should this be different?”
Yup, this is the point of Scott Alexander’s “MR Tries The Safe Uncertainty Fallacy”.
BTW, what “TC” is?
Tyler Cohen. I only used initials because the OP did the same.
And yes, I read that post, and I’ve seen similar arguments a number of times, and not just recently. They’re getting a lot sharper recently for obvious reasons, though.
TC is Tyler Cowen.
I don’t think the base rates are crazy—the new evolution of hominins one is only wrong if you forget who ‘you’ is. TC and many other people are assuming that ‘we’ will be the ‘you’ that are evolving. (The worry among people here is that ‘they’ will have their own ‘you’.)
And the second example, writing new software that breaks—that is the same as making any new technology, we have done this before, and we were fine last time. Yes there were computer viruses, yes some people lost fingers in looms back in the day. But it was okay in the long run.
I think people arguing against these base rates need to do more work. The base rates are reasonable, it is the lack of updating that makes the difference. So lets help them update!
I think updating against these base rates is the critical thing.
But it’s not really an update. The key difference between optimists and pessimists in this area is the recognition that there are no base rates for something like AGI. We have developed new technologies before, but we have never developed a new species before.
New forms of intelligence and agency are a completely new phenomena. Sonic you wanted to ascribe a base rate of our surviving this with zero previous examples, you’d put it at .5. if you counted all of the previous hominid extinctions as relevant, you’d actually put the base rate much lower.
This really seems like the relevant comparison. Tools don’t kill you, but strange creatures do. AGI will be a creature, not a tool.
See, the frustrating thing is, I really don’t think we are! There are loads of clear, concrete things that can be picked out and expanded upon. (See my sibling comment also.)
Honestly not sure if I agree or not, but even if true, it’s very hard to convince most people even with lots of real-world examples and data. Just ask anyone with an interest in the comparative quantitative risk assessment of different electricity sources, or ways of handling waste, and then ask them about the process of getting that permitted and built. And really, could you imagine if we subjected AI labs even to just 10% of the regulation we put in the way of letting people add a bedroom or bathroom to their houses?
Also, it doesn’t take a whole lot of abstraction to be more abstract than the physics examples I was responding too, and even then I don’t think we had nearly as much concrete data as we probably should have about the atmospheric nitrogen question. (Note that the H bomb developers also did the math that made them think Lithium-6 won’t contribute to yield, and were wrong. Not nearly as high stakes, so maybe they weren’t as careful? But still disconcerting).