First, this seems like an excellent issue to tackle, so I hope you get somewhere. This “fog of the future” objection is what stops me from taking MIRI more seriously. The obvious pattern matching of the UFAI with other apocalyptic scenarios does not help, either.
Second, when I ask myself “what argument/logic/experiment would convince me to take the AGI x-risk seriously enough to personally try to do something about it?”, I come up with nothing. Well, maybe a broad consensus among the AI researchers based on some solid experimental data, similarly to the current situation with anthropogenic climate change.
Just to make an extra step toward MIRI, suppose it had a convincing argument that without the FAI research the odds of human extinction due to UFAI are at least 10% (with high confidence), and that the FAI research can reduce the odds to, say, 1% (again, with high confidence), then I would possibly reevaluate my attitude.
I don’t see how any of the mentioned historical examples can do that. And definitely not any kind of counterfactual history scenarios, those have too low confidence to be taken seriously.
suppose [...] a convincing argument that without the [X] the odds of human extinction due to [Y] are at least 10% (with high confidence), and that the [X] can reduce the odds to, say, 1% (again, with high confidence), then I would possibly reevaluate my attitude.
I don’t think the hypothetical is true (by a large margin the expected impact is too big), but why only “possibly”? A high confidence intervention to avert a 9% risk of human extinction (for far less than 9% of world GDP) would be ludicrously good by normal standards.
Do you mean that “high confidence” is only conditional on the “convincing” argument, but “convincing” corresponds to relatively low confidence in the arguments itself? What is the hypothetical here?
I don’t think the hypothetical is true (by a large margin)
“A large margin” which way?
but why only “possibly”?
“Possibly” because:
I’d have to reevaluate the odds again, the confidence and my confidence in my confidence (probably no more meta than that) before actually changing my behavior based on that
compare with other potential x-risks prevention measures which can pop up at the same level of surprise when evaluated as thoroughly and at the same level
even if convinced that yes, AI indeed has a 10% or more chance of wiping out the human race as we know it AND would not replace it with something “better” in some sense of the word, AND that yes, MIRI can reduce this chance to mere 1%, AND no, other x-risk prevention efforts are not nearly as effective in improving the humans’ odds of surviving (in some form) the next century or millennium, I would also have to convince myself whether donating to MIRI and/or advocating for it, and/or volunteering and/or doing pro bono research for it would be an effective strategy.
Do you mean that “high confidence” is only conditional on the “convincing” argument, but “convincing” corresponds to relatively low confidence in the arguments itself? What is the hypothetical here?
Not sure I follow the question… I am no Bayesian, to me the argument being convincing is a statement about the odds of the argument being true, while the confidence in the predicted outcomes depends on how narrow the distribution the argument produces is, provided it’s true.
I see. I thought you were more in tune with Eliezer on this issue. I was simply trying to see what would make me take the MIRI research much more seriously. I am fascinated by the mathematical side of it, which is hopefully of high enough quality to attract expert attention, but I am currently much more skeptical of its effects on the odds of humanity surviving the next century or two.
I changed specifics to variables because I was interested more in the broader point than the specific case.
Asteroid tracking involved spending ~$100MM to eliminate most of the expected losses from civilization-wrecking asteroids. Generously, it might have eliminated as much as a 10^-6 extinction risk (if we had found a dinosaur-killer on course our civilization would have mobilized to divert it). At the same tradeoff, getting rid of a 9% extinction risk would seem to be worth $9T or more. Billions are spent on biodefense and nuclear nonproliferation programs each year.
So it seems to me that a 9% figure ‘overshoots’ the relevant thresholds in other areas: a much lower believed cost per increment of existential risk reduction would seem to suffice for more-than-adequate support (e.g. national governments, large foundations, and plenty of scientific talent would step in before that, based on experiences with nuclear weapons, climate change, cancer research, etc).
For comparison, consider someone who says that she will donate to malaria relief iff there is solidly convincing proof that at least 1000 cases of malaria affecting current people will be averted per dollar in the short-term. This is irrelevant in a world with a Gates Foundation, GiveWell, and so on: she will never get the chance as those with less stringent thresholds act.
I was trying to clarify whether you were using an extreme example to make the point in principle, or were saying that your threshold for action would actually be in that vicinity.
Second, when I ask myself “what argument/logic/experiment would convince me to take the AGI x-risk seriously enough to personally try to do something about it?”, I come up with nothing.
Is your position
AGI isn’t a a large x-risk
It’s too hard to do anything about it
or something else?
Just to make an extra step toward MIRI, suppose it had a convincing argument that without the FAI research the odds of human extinction due to UFAI are at least 10% (with high confidence), and that the FAI research can reduce the odds to, say, 1% (again, with high confidence), then I would possibly reevaluate my attitude.
You and I might be on the same page here. How broadly are you defining “FAI research” ?
I don’t see how any of the mentioned historical examples can do that. And definitely not any kind of counterfactual history scenarios, those have too low confidence to be taken seriously.
There are potentially promising interventions that are less targeted than the FAI research that MIRI is currently doing (e.g. lobbying for government regulations on AI research).
Can you clarify what sorts of counterfactual history scenarios you have in mind?
Is your position AGI isn’t a a large x-risk It’s too hard to do anything about it or something else?
I don’t have a well defended position. All I have is an estimate of confidence that my action or inaction would affect the hypothetical AGI x-risk in a known way. And that confidence is too low to be worth acting upon.
How broadly are you defining “FAI research” ?
Any research included in such an argument, in any area. Really, anything that provides some certainty.
There are potentially promising interventions that are less targeted than the FAI research that MIRI is currently doing (e.g. lobbying for government regulations on AI research).
I have extremely low confidence that these interventions can affect the hypothetical AGI x-risk in the desired direction.
Can you clarify what sorts of counterfactual history scenarios you have in mind?
I can’t imagine anything convincing. Similarly, I don’t find an argument “if one of the Hitler assassination attempts were successful, would be avoided” compelling. Not to say that one should not have tried to assassinate him at the time, given the information available. But a valid reason to carry out such an assassination attempt would have to be something near-term and high-confidence, like reducing the odds of further poor military decisions or something.
I don’t have a well defended position. All I have is an estimate of confidence that my action or inaction would affect the hypothetical AGI x-risk in a known way. And that confidence is too low to be worth acting upon.
This is close to my current position, but I would update if I learned that there’s a non-negligible chance of AGI within the next 20 years.
I have extremely low confidence that these interventions can affect the hypothetical AGI x-risk in the desired direction.
This is the issue under investigation
I can’t imagine anything convincing. Similarly, I don’t find an argument “if one of the Hitler assassination attempts were successful, would be avoided” compelling. Not to say that one should not have tried to assassinate him at the time, given the information available. But a valid reason to carry out such an assassination attempt would have to be something near-term and high-confidence, like reducing the odds of further poor military decisions or something.
What about policies to reduce hydrofluorocarbons emissions that would otherwise deplete the ozone layer?
First, this seems like an excellent issue to tackle, so I hope you get somewhere. This “fog of the future” objection is what stops me from taking MIRI more seriously. The obvious pattern matching of the UFAI with other apocalyptic scenarios does not help, either.
Second, when I ask myself “what argument/logic/experiment would convince me to take the AGI x-risk seriously enough to personally try to do something about it?”, I come up with nothing. Well, maybe a broad consensus among the AI researchers based on some solid experimental data, similarly to the current situation with anthropogenic climate change.
Just to make an extra step toward MIRI, suppose it had a convincing argument that without the FAI research the odds of human extinction due to UFAI are at least 10% (with high confidence), and that the FAI research can reduce the odds to, say, 1% (again, with high confidence), then I would possibly reevaluate my attitude.
I don’t see how any of the mentioned historical examples can do that. And definitely not any kind of counterfactual history scenarios, those have too low confidence to be taken seriously.
I don’t think the hypothetical is true (by a large margin the expected impact is too big), but why only “possibly”? A high confidence intervention to avert a 9% risk of human extinction (for far less than 9% of world GDP) would be ludicrously good by normal standards.
Do you mean that “high confidence” is only conditional on the “convincing” argument, but “convincing” corresponds to relatively low confidence in the arguments itself? What is the hypothetical here?
“A large margin” which way?
“Possibly” because:
I’d have to reevaluate the odds again, the confidence and my confidence in my confidence (probably no more meta than that) before actually changing my behavior based on that
compare with other potential x-risks prevention measures which can pop up at the same level of surprise when evaluated as thoroughly and at the same level
even if convinced that yes, AI indeed has a 10% or more chance of wiping out the human race as we know it AND would not replace it with something “better” in some sense of the word, AND that yes, MIRI can reduce this chance to mere 1%, AND no, other x-risk prevention efforts are not nearly as effective in improving the humans’ odds of surviving (in some form) the next century or millennium, I would also have to convince myself whether donating to MIRI and/or advocating for it, and/or volunteering and/or doing pro bono research for it would be an effective strategy.
Not sure I follow the question… I am no Bayesian, to me the argument being convincing is a statement about the odds of the argument being true, while the confidence in the predicted outcomes depends on how narrow the distribution the argument produces is, provided it’s true.
9% is far too high.
I see. I thought you were more in tune with Eliezer on this issue. I was simply trying to see what would make me take the MIRI research much more seriously. I am fascinated by the mathematical side of it, which is hopefully of high enough quality to attract expert attention, but I am currently much more skeptical of its effects on the odds of humanity surviving the next century or two.
I changed specifics to variables because I was interested more in the broader point than the specific case.
Asteroid tracking involved spending ~$100MM to eliminate most of the expected losses from civilization-wrecking asteroids. Generously, it might have eliminated as much as a 10^-6 extinction risk (if we had found a dinosaur-killer on course our civilization would have mobilized to divert it). At the same tradeoff, getting rid of a 9% extinction risk would seem to be worth $9T or more. Billions are spent on biodefense and nuclear nonproliferation programs each year.
So it seems to me that a 9% figure ‘overshoots’ the relevant thresholds in other areas: a much lower believed cost per increment of existential risk reduction would seem to suffice for more-than-adequate support (e.g. national governments, large foundations, and plenty of scientific talent would step in before that, based on experiences with nuclear weapons, climate change, cancer research, etc).
For comparison, consider someone who says that she will donate to malaria relief iff there is solidly convincing proof that at least 1000 cases of malaria affecting current people will be averted per dollar in the short-term. This is irrelevant in a world with a Gates Foundation, GiveWell, and so on: she will never get the chance as those with less stringent thresholds act.
I was trying to clarify whether you were using an extreme example to make the point in principle, or were saying that your threshold for action would actually be in that vicinity.
Is your position
AGI isn’t a a large x-risk
It’s too hard to do anything about it
or something else?
You and I might be on the same page here. How broadly are you defining “FAI research” ?
There are potentially promising interventions that are less targeted than the FAI research that MIRI is currently doing (e.g. lobbying for government regulations on AI research).
Can you clarify what sorts of counterfactual history scenarios you have in mind?
I don’t have a well defended position. All I have is an estimate of confidence that my action or inaction would affect the hypothetical AGI x-risk in a known way. And that confidence is too low to be worth acting upon.
Any research included in such an argument, in any area. Really, anything that provides some certainty.
I have extremely low confidence that these interventions can affect the hypothetical AGI x-risk in the desired direction.
I can’t imagine anything convincing. Similarly, I don’t find an argument “if one of the Hitler assassination attempts were successful, would be avoided” compelling. Not to say that one should not have tried to assassinate him at the time, given the information available. But a valid reason to carry out such an assassination attempt would have to be something near-term and high-confidence, like reducing the odds of further poor military decisions or something.
This is close to my current position, but I would update if I learned that there’s a non-negligible chance of AGI within the next 20 years.
This is the issue under investigation
What about policies to reduce hydrofluorocarbons emissions that would otherwise deplete the ozone layer?
Well, there is no need for any fancy counterfactual history there, the link was confirmed experimentally with high confidence.
Yes the Montreal Protocol, an extremely successful international treaty.
By the way, do I know you personally? Feel free to email me at jsinick@gmail.com if you’d like to correspond.
I doubt it. And I don’t think I have much to contribute to any genuine AGI/risk research.