Suppose pro-friendly AI and anti-uncontrolled-AI advocacy and research is not at this point the most effective mitigation of x-risk.
That’s not the problem. The problem is that it’s not mitigation at all, it’s exacerbation. The current state of affairs is not stable (for that matter, it’s not even in equilibrium); either we go up or we go down. If we snuff out real research in favor of hopeless feel-good programs to formalize Friendliness with pen and paper, we throw away chances of the former and thereby choose the latter by default.
Remember, it is the way of extinction that what kills the last individual often has nothing to do with the factors that doomed the species. For all anyone knows, the last dodo may have died of old age. I’m confident there will still be at least some people alive in 2100. But whether there still exists a winning move for humanity at that stage may depend on what we choose to support now, in the early decades of the century.
I grant that if it (thinking about FAI) were certain to be harmful, then absolutely none of it should be done. I didn’t even consider that possibility.
I don’t think it’s certainly harmful, and I believe it has some expected (or at least diversification) benefit.
Remember, it is the way of extinction that what kills the last individual often has nothing to do with the factors that doomed the species. For all anyone knows, the last dodo may have died of old age. I’m confident there will still be at least some people alive in 2100. But whether there still exists a winning move for humanity at that stage may depend on what we choose to support now, in the early decades of the century.
What if we take a survivalist-type approach, putting a bunch of people and natural resources deep underground somewhere?
That would protect against certain straightforward kinds of disaster e.g. asteroid impact, but not against more subtle and more likely threats. Remember, people in a deep underground shelter will still die of old age just as they would have on the surface.
I think we’re better off spending the resources more proactively, unless and until we find evidence of an imminent threat of a variety against which that is a good defense. For example, instead of spending money preemptively populating underground shelters in case of an asteroid impact, I’d rather spend it extending our surveys of the sky to have a better chance of spotting an incoming asteroid in time to deflect it.
This depends on being able to anticipate all significant threats.
But I guess it’s silly to debate what percentage of all humanity’s resources should be devoted to various things—just what percentage of our resources.
Agreed on both counts; of course we have no way to know exactly what threats we face, let alone exactly how to deal with them.
Except in this regard: however long or short our window of opportunity may be, any slowdown in technological progress increases the general threat that some specific threat will close that window on us while we are still vulnerable, while we still depend on nonrenewable resources, while all our eggs are still in one basket. The one form of protection we need above all else is speed, and that’s how I believe we should be spending as much as possible of our resources.
That’s not the problem. The problem is that it’s not mitigation at all, it’s exacerbation. The current state of affairs is not stable (for that matter, it’s not even in equilibrium); either we go up or we go down. If we snuff out real research in favor of hopeless feel-good programs to formalize Friendliness with pen and paper, we throw away chances of the former and thereby choose the latter by default.
Remember, it is the way of extinction that what kills the last individual often has nothing to do with the factors that doomed the species. For all anyone knows, the last dodo may have died of old age. I’m confident there will still be at least some people alive in 2100. But whether there still exists a winning move for humanity at that stage may depend on what we choose to support now, in the early decades of the century.
I grant that if it (thinking about FAI) were certain to be harmful, then absolutely none of it should be done. I didn’t even consider that possibility.
I don’t think it’s certainly harmful, and I believe it has some expected (or at least diversification) benefit.
What if we take a survivalist-type approach, putting a bunch of people and natural resources deep underground somewhere?
That would protect against certain straightforward kinds of disaster e.g. asteroid impact, but not against more subtle and more likely threats. Remember, people in a deep underground shelter will still die of old age just as they would have on the surface.
OK, so what percentage of humanity’s resources (human and natural) do you think should be kept in reserve underground?
My guess is we both agree it should be way more than it is right now.
Research “sex” :-P
I think we’re better off spending the resources more proactively, unless and until we find evidence of an imminent threat of a variety against which that is a good defense. For example, instead of spending money preemptively populating underground shelters in case of an asteroid impact, I’d rather spend it extending our surveys of the sky to have a better chance of spotting an incoming asteroid in time to deflect it.
This depends on being able to anticipate all significant threats.
But I guess it’s silly to debate what percentage of all humanity’s resources should be devoted to various things—just what percentage of our resources.
Agreed on both counts; of course we have no way to know exactly what threats we face, let alone exactly how to deal with them.
Except in this regard: however long or short our window of opportunity may be, any slowdown in technological progress increases the general threat that some specific threat will close that window on us while we are still vulnerable, while we still depend on nonrenewable resources, while all our eggs are still in one basket. The one form of protection we need above all else is speed, and that’s how I believe we should be spending as much as possible of our resources.