It could instead be “there is a real threat model for existential risk, and it is important that society does more to address it than it is currently doing”. If you’re going to share concerns at all, figure out the position you do have courage in, and then discuss that as if it is obvious and sensible, not as if you are ashamed of it.
The potential failure mode I see with this is that, if you’re not paying sufficient attention to your rhetoric, you run the risk of activating people’s anti-Pascal’s mugging instincts.[1] As Jeremy Gillen said:
Everyone is pretty frequently bombarded with wild arguments and beliefs that have crazy implications. Like conspiracy theories, political claims, spiritual claims, get-rich-quick schemes, scientific discoveries, news headlines, mental health and wellness claims, alternative medicine, claims about which lifestyles are better. We don’t often have the time (nor expertise or skill or sometimes intelligence) to evaluate them properly.
“You should pay attention to this even if you think there’s a really small chance of it happening because if it does happen, the consequences will be massive” is something most people hear pretty regularly. In lieu of spending inordinate amounts of time teasing out the specific probabilities and expected utilities, they create heuristics allowing them to ignore these kinds of claims.
Note: in my experience obtained from reading Substack comments on posts by EA-skeptical authors, the belief that “EA wants you to care about AI safety because there’s a low probability of a really bad outcome” is both extremely prevalent and also causes EA/AI x-risk proponents/etc to be viewed quite negatively (like they’re using argumentative Dark Arts or something similar).
I don’t think most anyone who’s studied the issues at hand thinks the chance of danger is “really small”, even among people who disagree with me quite a lot (see e.g. here). I think folks who retreat to arguments like “you should pay attention to this even if you think there’s a really small chance of it happening” are doing a bunch of damage, and this is one of many problems I attribute to a lack of this “courage” stuff I’m trying to describe.
When I speak of “finding a position you have courage in”, I do not mean “find a position that you think should be logically unassailable.” I’m apparently not doing a very good job at transmitting the concept, but here’s some positive and negative examples:
✓ “The race towards superintelligence is ridiculously risky and I don’t think humanity should be doing it.”
✓ “I’m not talking about a tiny risk. On my model, this is your most likely cause of death.”
✓ “Oh I think nobody should be allowed to race towards superintelligence, but I’m trying to build it anyway because I think I’ll do it better than the next guy. Ideally all AI companies in the world should be shut down, though, because we’d need way more time to do this properly.” (The action is perhaps a bit cowardly, but the statement is courageous, if spoken by someone for whom it’s true.)
✗ “Well we can all agree that it’d be bad if AIs were used to enable terrorists to make bioweapons” (spoken by someone who thinks the danger from superintelligence is pressing).
✗ “Even if you think the chance of it happening is very small, it’s worth focusing on, because the consequences are so huge” (spoken by someone who believes the danger is substantial).
✗ “In some unlikely but extreme cases, these companies put civilization at risk, and the companies should be responsible for managing those tail risks” (spoken by someone who believes the danger is substantial).
One litmus test here is: have you communicated the real core of the situation and its direness as you perceive it? Not like “have you caused them as much concern as you can manage to”, more like “have you actually just straightforwardly named the key issue”. (There’s also some caveats here about how, if you think there’s a lowish chance of disaster because you think humanity will come to its senses and change course, then this notion of “courage” I’m trying to name still entails communicating how humanity is currently on course for a full-fledged disaster, without mincing words.)
Something I notice is that in the good examples you use only I statements. “I don’t think humanity should be doing it”, “I’m not talking about a tiny risk”, “Oh I think I’ll do it better than the next guy”.
Whereas in the bad examples it’s different, “Well we can all agree that it’d be bad if AIs were used to enable terrorists to make bioweapons”, “Even if you think the chance of it happening is very small”, “In some unlikely but extreme cases, these companies put civilization at risk”
I think with the bad examples there’s a lot of pressure for the other person to agree, “the companies should be responsible (because I say so)”, “Even if you think… Its still worth focusing on (because I’ve decided what you should care about)”, “Well we can all agree (I’ve already decided you agree and you’re not getting a choice otherwise)”
Whereas with the good examples the other person is not under any pressure to agree, so they are completely free to think about the things you’re saying. I think that’s also part of what makes these statements courageous, that it’s stated in a way where the other person is free to agree or dissagree as they wish, and so you trust that what your saying is compelling enough to be persuasive on its own.
I don’t think most anyone who’s studied the issues at hand thinks the chance of danger is “really small”, even among people who disagree with me quite a lot (see e.g. here)
This link doesn’t seem to include people like Quintin Pope and the AI Optimists, who are the most notorious AI risk skeptics I can think of who have nonetheless written about Eliezer’s arguments (example). If I recall correctly, I think Pope said sometime before his departure from this site that his P(doom) is around 1%.
Yup, that claim is wrong. I’m not ⇐ 1% but I have met educated skeptics who are. Not sure why Nate made this claim since it isn’t relevant to his point—could just delete that first sentence.
in my experience obtained from reading Substack comments on posts by EA-skeptical authors, the belief that “EA wants you to care about AI safety because there’s a low probability of a really bad outcome” is both extremely prevalent and also causes EA/AI x-risk proponents/etc to be viewed quite negatively (like they’re using argumentative Dark Arts or something similar).
I have a similar experience from reading Hacker News. Seems to me that people who write it don’t really want to express an opinion on AI, more like they are using absurdity heuristic for an attack by association against EA. (Attacking EA is their goal, arguing by “AI apocalypse is unlikely” is merely a convenient weapon.)
The potential failure mode I see with this is that, if you’re not paying sufficient attention to your rhetoric, you run the risk of activating people’s anti-Pascal’s mugging instincts.[1] As Jeremy Gillen said:
“You should pay attention to this even if you think there’s a really small chance of it happening because if it does happen, the consequences will be massive” is something most people hear pretty regularly. In lieu of spending inordinate amounts of time teasing out the specific probabilities and expected utilities, they create heuristics allowing them to ignore these kinds of claims.
Note: in my experience obtained from reading Substack comments on posts by EA-skeptical authors, the belief that “EA wants you to care about AI safety because there’s a low probability of a really bad outcome” is both extremely prevalent and also causes EA/AI x-risk proponents/etc to be viewed quite negatively (like they’re using argumentative Dark Arts or something similar).
I don’t think most anyone who’s studied the issues at hand thinks the chance of danger is “really small”, even among people who disagree with me quite a lot (see e.g. here). I think folks who retreat to arguments like “you should pay attention to this even if you think there’s a really small chance of it happening” are doing a bunch of damage, and this is one of many problems I attribute to a lack of this “courage” stuff I’m trying to describe.
When I speak of “finding a position you have courage in”, I do not mean “find a position that you think should be logically unassailable.” I’m apparently not doing a very good job at transmitting the concept, but here’s some positive and negative examples:
✓ “The race towards superintelligence is ridiculously risky and I don’t think humanity should be doing it.”
✓ “I’m not talking about a tiny risk. On my model, this is your most likely cause of death.”
✓ “Oh I think nobody should be allowed to race towards superintelligence, but I’m trying to build it anyway because I think I’ll do it better than the next guy. Ideally all AI companies in the world should be shut down, though, because we’d need way more time to do this properly.” (The action is perhaps a bit cowardly, but the statement is courageous, if spoken by someone for whom it’s true.)
✗ “Well we can all agree that it’d be bad if AIs were used to enable terrorists to make bioweapons” (spoken by someone who thinks the danger from superintelligence is pressing).
✗ “Even if you think the chance of it happening is very small, it’s worth focusing on, because the consequences are so huge” (spoken by someone who believes the danger is substantial).
✗ “In some unlikely but extreme cases, these companies put civilization at risk, and the companies should be responsible for managing those tail risks” (spoken by someone who believes the danger is substantial).
One litmus test here is: have you communicated the real core of the situation and its direness as you perceive it? Not like “have you caused them as much concern as you can manage to”, more like “have you actually just straightforwardly named the key issue”. (There’s also some caveats here about how, if you think there’s a lowish chance of disaster because you think humanity will come to its senses and change course, then this notion of “courage” I’m trying to name still entails communicating how humanity is currently on course for a full-fledged disaster, without mincing words.)
Something I notice is that in the good examples you use only I statements. “I don’t think humanity should be doing it”, “I’m not talking about a tiny risk”, “Oh I think I’ll do it better than the next guy”.
Whereas in the bad examples it’s different, “Well we can all agree that it’d be bad if AIs were used to enable terrorists to make bioweapons”, “Even if you think the chance of it happening is very small”, “In some unlikely but extreme cases, these companies put civilization at risk”
I think with the bad examples there’s a lot of pressure for the other person to agree, “the companies should be responsible (because I say so)”, “Even if you think… Its still worth focusing on (because I’ve decided what you should care about)”, “Well we can all agree (I’ve already decided you agree and you’re not getting a choice otherwise)”
Whereas with the good examples the other person is not under any pressure to agree, so they are completely free to think about the things you’re saying. I think that’s also part of what makes these statements courageous, that it’s stated in a way where the other person is free to agree or dissagree as they wish, and so you trust that what your saying is compelling enough to be persuasive on its own.
This link doesn’t seem to include people like Quintin Pope and the AI Optimists, who are the most notorious AI risk skeptics I can think of who have nonetheless written about Eliezer’s arguments (example). If I recall correctly, I think Pope said sometime before his departure from this site that his P(doom) is around 1%.
Yup, that claim is wrong. I’m not ⇐ 1% but I have met educated skeptics who are. Not sure why Nate made this claim since it isn’t relevant to his point—could just delete that first sentence.
(The existence of exceptions is why I said “most anyone” instead of “anyone”.)
I have a similar experience from reading Hacker News. Seems to me that people who write it don’t really want to express an opinion on AI, more like they are using absurdity heuristic for an attack by association against EA. (Attacking EA is their goal, arguing by “AI apocalypse is unlikely” is merely a convenient weapon.)