The standard answer here is that all humans dying is much, much worse than anything happening with narrow AI. Not to say those problems are trivial, but humanity’s extinction is an entirely different level of bad, so that’s what we should be focusing on. This is even more true if you care about future generations, since human extinction is not just 7 billion dead, but the loss of all generations who could have come after.
I personally believe this argument holds even if we ascribe a relatively low probability to AGI in the relatively near future. E.g, if you think there’s a 10% chance of AGI in the next 10-20 years, it still seems reasonable to prioritise AGI safety now. If you think AGI isn’t possible at all, naturally we don’t need to worry about AI safety. But I find that pretty unconvincing—humanity has made a lot of progress very quickly in the field of AI capabilities, and it shows no signs of slowing down, and there’s no reason why such a machine could not exist in principle.
I understand and appreciate your discussion. I wonder if perhaps we could consider is that it may be more morally imperative to work on AI safety for the hugely impactful problems AI is contributing right now, if we assume that in finding solutions to these current and near-term AI problems we would also be lowering the risk of AGI X-risk (albiet indirectly).
Given that the likelyhood for narrow AI risk being 1 and the likelyhood of AGI in the next 10 years being (as in your example) <0.1 - It seems obvious we should focus on addressing the former as not only will it reduce suffering that we know with certainity is already happening but also suffer that will certainly continue to happen, in addition it will also indirectly reduce X-risk. If we combine this observation with the opportunity cost in not solving other even more solvable issues (disease, education etc). It seems even less appealing to pour millions of dollars and the careers of the smartest people in specifically AGI X-Risk.
A final point, is that it would seem the worst issues caused by current and near-term AI risks are that it is degrading the coordination structures of western democracies. (Disinformation, polarisation and so on). If, following Moloch, we understand coordination to be the most vital tool in humanity’s adressing of problems we see that focusing on current AI safety issues will improve our ability of addressing every other area of human suffering.
The opportunity costs in not focusing on improving coordination problems in western countries seem to be equivalent to x-risk level consequences, while the probability of the first is 1 and that of AGI >1.
If you consider these coordination problems to be equivalent to x-risk level consequences, then it makes sense to work on aligning narrow AI. For instance, if you think there’s a 10% chance of AGI x-risk this century, and current problems are 10% as bad as human extinction. After all, working on aligning narrow AI is probably more tractable than working on aligning the hypothetical AGI systems of the future. You are also right that aligning narrow AI may help align AGI in the future—it is, at the very least, unlikely to hurt.
Personally, I don’t think the current problems are anything close to “10% as bad as human extinction”, but you may disagree with me on this. I’m not very knowledgable about politics, which is the field I would look into to try and identify the harms caused by our current degradation of coordination, so I’m not going to try to convince you of anything in that field—more trying to provide a framework with which to look at potential problems.
So, basically I would look at it as—which is higher? The chance of human extinction from AGI times the consequences? Or the x-risk reduction from aligning narrow AI, plus the positive utility of solving our current problems today? I believe the former, so I think AGI is more important. If you believe the latter, aligning narrow AI is more important.
Intersting, yes I am interested in coordination problems. Let me follow this framework, to make a better case. There are three considerations I would like to point out.
The utility in adressing coordination problems is that they affect almost all X-risk scenarios.(Nuclear war, bioterror, pandemics, climate change and AGI) Working on coordintion problems reduces not only current suffering but also X-Risk of both AGI and non AGI kinds.
The difference between a 10% chance of something happening that may be an X-Risk in 100 years is not 10 times less then something with a certainity of happening. Its not even comparable because one is a certainity the other a probability, and we only get one roll of the dice (allocation of resources). It seems that the rational choice would always be the certainity.
More succinctly, with two buttons one with a 100% chance of adressing X-Risk and one with a 10% chance, which one would you press?
As for Point 2, you can absolutely compare a certainty and a probability. If I offered you a certainty of $10, or a 10% chance of $1,000,000, would you take the $10 because you can’t compare certainties and probabilities, and I’m only ever going to offer you the deal once?
That then brings me to question 3. The button I would press would be the one that reduces total X-risk the most. If both buttons reduced X-risk by 1%, I would press the 100% one. If the 100% button reduced X-risk by 0.1%, and the 10% one reduced X-risk by 10%, I would pick the second one, for an expected value of 1% X-risk reduction. You have to take the effect size into account. We can disagree on what the effect sizes are, but you still need to consider them.
Interesting, I see what you mean reagrding probability and it makes sense. I guess perhaps, what is missing is that when it comes to questions of peoples lives we may have a stronger imperative to be more risk-averse.
I completely agree with you about effect size. I guess what I would say is that that given my point 1 from earlier about the variety of X-risks coordination would contirbute in solving then the effect size will always be greater. If we want to maximise utility its the best chance we have. The added bonuses are that it is comparatively tractable and immediate avoiding the recent criticicisms about longtermism, while simoultnously being a longtermist solution.
Regadless, it does seem that coordination problems are underdiscussed in the community, will try and make a decent main post once my academic committments clear up a bit.
Being risk-averse around people’s lives is only a good strategy when you’re trading off against something else that isn’t human lives. If you have the choice to save 400 lives with certainty, or a 90% chance to save 500 lives, choosing the former is essentially condemning 50 people to death. At that point, you’re just behaving suboptimally.
Being risk-averse works if you’re trading off other things. E.g, if you could release a new car now that you’re almost certain is safe, you might be risk-averse and call for more tests. As long as people won’t die from you delaying this car, being risk-averse is a reasonable strategy here.
Given your Point 1 from earlier, there is no reason to expect the effect size will always be greater. If the effect on reducing X-risks from co-ordination becomes small enough, or the risk of a particular X-risk becomes large enough, this changes the equation. If you believe, like many in this forum do, that AI represents the lion’s share of X-risk, focusing on AI directly is probably more effective. If you believe that x-risk is diversified, that there’s some chance from AI, some from pandemics, some from nuclear war, some from climate change, etc. then co-ordination makes more sense. Co-ordination has a small effect on all x-risks, direct work has a larger effect on a single x-risk.
The point I’m trying to make here is this. There are perfectly reasonable states of the world where “Improve co-ordination” is the best action to take to reduce x-risk. There are also perfectly reasonable states of the world where “Work directly on <Risk A>” is the best action to take to reduce x-risk. You won’t be able to find out which is which if you believe one is “always” the case.
What I would suggest is to ask “What would cause me to change my mind and believe improving co-ordination is NOT the best way to work on x-risk”, and then seek out whether those things are true or not. If you don’t believe they are, great, that’s fine.
That said, it wouldn’t be fair to ask you what would change your mind without presenting my own. On my end, what would convince me that improving co-ordination is more important than direct AI work:
AI is less dangerous than I expect, so that the x-risk profile is more diversified instead of mostly AI.
We already have more technical progress in AI safety than I believe we have, so we don’t need more and should focus on either co-ordination or the next most dangerous x-risk. (Which I believe is pandemics, which is both less dangerous than AI and more responsive to government co-ordination in my opinion)
AI is far more dangerous than I expect, to the point where the AI alignment problem is unsolvable and co-ordination is the only solution.
The standard answer here is that all humans dying is much, much worse than anything happening with narrow AI. Not to say those problems are trivial, but humanity’s extinction is an entirely different level of bad, so that’s what we should be focusing on. This is even more true if you care about future generations, since human extinction is not just 7 billion dead, but the loss of all generations who could have come after.
I personally believe this argument holds even if we ascribe a relatively low probability to AGI in the relatively near future. E.g, if you think there’s a 10% chance of AGI in the next 10-20 years, it still seems reasonable to prioritise AGI safety now. If you think AGI isn’t possible at all, naturally we don’t need to worry about AI safety. But I find that pretty unconvincing—humanity has made a lot of progress very quickly in the field of AI capabilities, and it shows no signs of slowing down, and there’s no reason why such a machine could not exist in principle.
I understand and appreciate your discussion. I wonder if perhaps we could consider is that it may be more morally imperative to work on AI safety for the hugely impactful problems AI is contributing right now, if we assume that in finding solutions to these current and near-term AI problems we would also be lowering the risk of AGI X-risk (albiet indirectly).
Given that the likelyhood for narrow AI risk being 1 and the likelyhood of AGI in the next 10 years being (as in your example) <0.1 - It seems obvious we should focus on addressing the former as not only will it reduce suffering that we know with certainity is already happening but also suffer that will certainly continue to happen, in addition it will also indirectly reduce X-risk. If we combine this observation with the opportunity cost in not solving other even more solvable issues (disease, education etc). It seems even less appealing to pour millions of dollars and the careers of the smartest people in specifically AGI X-Risk.
A final point, is that it would seem the worst issues caused by current and near-term AI risks are that it is degrading the coordination structures of western democracies. (Disinformation, polarisation and so on). If, following Moloch, we understand coordination to be the most vital tool in humanity’s adressing of problems we see that focusing on current AI safety issues will improve our ability of addressing every other area of human suffering.
The opportunity costs in not focusing on improving coordination problems in western countries seem to be equivalent to x-risk level consequences, while the probability of the first is 1 and that of AGI >1.
If you consider these coordination problems to be equivalent to x-risk level consequences, then it makes sense to work on aligning narrow AI. For instance, if you think there’s a 10% chance of AGI x-risk this century, and current problems are 10% as bad as human extinction. After all, working on aligning narrow AI is probably more tractable than working on aligning the hypothetical AGI systems of the future. You are also right that aligning narrow AI may help align AGI in the future—it is, at the very least, unlikely to hurt.
Personally, I don’t think the current problems are anything close to “10% as bad as human extinction”, but you may disagree with me on this. I’m not very knowledgable about politics, which is the field I would look into to try and identify the harms caused by our current degradation of coordination, so I’m not going to try to convince you of anything in that field—more trying to provide a framework with which to look at potential problems.
So, basically I would look at it as—which is higher? The chance of human extinction from AGI times the consequences? Or the x-risk reduction from aligning narrow AI, plus the positive utility of solving our current problems today? I believe the former, so I think AGI is more important. If you believe the latter, aligning narrow AI is more important.
Intersting, yes I am interested in coordination problems. Let me follow this framework, to make a better case. There are three considerations I would like to point out.
The utility in adressing coordination problems is that they affect almost all X-risk scenarios.(Nuclear war, bioterror, pandemics, climate change and AGI) Working on coordintion problems reduces not only current suffering but also X-Risk of both AGI and non AGI kinds.
The difference between a 10% chance of something happening that may be an X-Risk in 100 years is not 10 times less then something with a certainity of happening. Its not even comparable because one is a certainity the other a probability, and we only get one roll of the dice (allocation of resources). It seems that the rational choice would always be the certainity.
More succinctly, with two buttons one with a 100% chance of adressing X-Risk and one with a 10% chance, which one would you press?
I agree with you on the first point completely.
As for Point 2, you can absolutely compare a certainty and a probability. If I offered you a certainty of $10, or a 10% chance of $1,000,000, would you take the $10 because you can’t compare certainties and probabilities, and I’m only ever going to offer you the deal once?
That then brings me to question 3. The button I would press would be the one that reduces total X-risk the most. If both buttons reduced X-risk by 1%, I would press the 100% one. If the 100% button reduced X-risk by 0.1%, and the 10% one reduced X-risk by 10%, I would pick the second one, for an expected value of 1% X-risk reduction. You have to take the effect size into account. We can disagree on what the effect sizes are, but you still need to consider them.
Interesting, I see what you mean reagrding probability and it makes sense. I guess perhaps, what is missing is that when it comes to questions of peoples lives we may have a stronger imperative to be more risk-averse.
I completely agree with you about effect size. I guess what I would say is that that given my point 1 from earlier about the variety of X-risks coordination would contirbute in solving then the effect size will always be greater. If we want to maximise utility its the best chance we have. The added bonuses are that it is comparatively tractable and immediate avoiding the recent criticicisms about longtermism, while simoultnously being a longtermist solution.
Regadless, it does seem that coordination problems are underdiscussed in the community, will try and make a decent main post once my academic committments clear up a bit.
Being risk-averse around people’s lives is only a good strategy when you’re trading off against something else that isn’t human lives. If you have the choice to save 400 lives with certainty, or a 90% chance to save 500 lives, choosing the former is essentially condemning 50 people to death. At that point, you’re just behaving suboptimally.
Being risk-averse works if you’re trading off other things. E.g, if you could release a new car now that you’re almost certain is safe, you might be risk-averse and call for more tests. As long as people won’t die from you delaying this car, being risk-averse is a reasonable strategy here.
Given your Point 1 from earlier, there is no reason to expect the effect size will always be greater. If the effect on reducing X-risks from co-ordination becomes small enough, or the risk of a particular X-risk becomes large enough, this changes the equation. If you believe, like many in this forum do, that AI represents the lion’s share of X-risk, focusing on AI directly is probably more effective. If you believe that x-risk is diversified, that there’s some chance from AI, some from pandemics, some from nuclear war, some from climate change, etc. then co-ordination makes more sense. Co-ordination has a small effect on all x-risks, direct work has a larger effect on a single x-risk.
The point I’m trying to make here is this. There are perfectly reasonable states of the world where “Improve co-ordination” is the best action to take to reduce x-risk. There are also perfectly reasonable states of the world where “Work directly on <Risk A>” is the best action to take to reduce x-risk. You won’t be able to find out which is which if you believe one is “always” the case.
What I would suggest is to ask “What would cause me to change my mind and believe improving co-ordination is NOT the best way to work on x-risk”, and then seek out whether those things are true or not. If you don’t believe they are, great, that’s fine.
That said, it wouldn’t be fair to ask you what would change your mind without presenting my own. On my end, what would convince me that improving co-ordination is more important than direct AI work:
AI is less dangerous than I expect, so that the x-risk profile is more diversified instead of mostly AI.
We already have more technical progress in AI safety than I believe we have, so we don’t need more and should focus on either co-ordination or the next most dangerous x-risk. (Which I believe is pandemics, which is both less dangerous than AI and more responsive to government co-ordination in my opinion)
AI is far more dangerous than I expect, to the point where the AI alignment problem is unsolvable and co-ordination is the only solution.