I think there is some way that the conversation needs to advance, and I think this is roughly carving at some real joints and it’s important that people are tracking the distinction.
But
a) I’m generally worried about reifying the groups more into existence (as opposed to trying to steer towards a world where people can have more nuanced views). This is tricky, there are tradeoffs and I’m not sure how to handle this. But...
b) this post title and framing particular is super leaning into the polarization and I wish it did something different.
I don’t like polarization as such, but I also don’t like all of my loved ones being killed. I see this post and the open statement as dissolving a conflationary alliance that groups people who want to (at least temporarily) prevent the creation of superintelligence with people who don’t want to do that. Those two groups of people are trying to do very different things that I expect will have very different outcomes.
I don’t think the people in Camp A are immoral people just for holding that position[1], but I do think it is necessary to communicate: “If we do thing A, we will die. You must stop trying to do thing A, because that will kill everyone. Thing B will not kill everyone. These are not the same thing.”
In general, to actually get the things that you want in the world, sometimes you have to fight very hard for them, even against other people. Sometimes you have to optimize for convincing people. Sometimes you have to shame people. The norms of discourse that are comfortable for me and elevate truth-seeking and that make LessWrong a wonderful place are not always the same patterns as those that are most likely to cause us and our families to still be alive in the near future.
Though I have encountered some people in the AI Safety community who are happy to unnecessarily subject others to extreme risks without their consent after a naive utilitarian calculus on their behalf, which I do consider grossly immoral.
Sure—but are they in this case different? Is the polarization framing worth it here? I don’t think so, because polarization has large downsides.
You can argue against the “race to AGI” position without making the division into camps worse. Calling people kind of stupid for holding the position they do (which Tegmark’s framing definitely does) is a way to get people to dig in, double down, and fight you instead of quietly shifting their views based on logic.
I agree with him that position A is illogical; I think any reasonable estimate of risks demands more caution than the position he lays out. And I think that many people actually hold that position. But many have more nuanced views, which could firm up into a group identity and otherwise fairly risk-aware people putting effort into arguing against strong risk arguments based on defensive emotional responses.
Fine, and also I’m not saying what to do about it (shame or polarize or whatever), but PRIOR to that, we have to STOP PRETENDING IT’S JUST A VIEW. It’s a conflictual stance that they are taking. It’s like saying that the statisticians arguing against “smoking causes cancer” “have a nuanced view”.
I’m not pretending it’s just a view. The immense importance of this issue is another reason to avoid polarization. Look at how the climate change issue worked out with polarization involved.
The arguments for caution are very strong. Proponents of caution are at an advantage in a discussion. We’re also a minority, so we’re at a disadvantage in a fight. So it seems important to not help it move from being a discussion to a fight.
climate change issue worked out with polarization involved
The climate change issue has pretty widespread international agreement and in most countries is considered a bipartisan issue. The capture of climate change by polarising forces has not really affected intervention outcomes (other problems of implementation are, imo, far greater).
I don’t want to derail the AI safety organising conversation, but I see this climate change comparison come up a lot. It strikes me as a pretty low-quality argument and it’s not clear a) whether the central claim is even true and b) whether it is transferable to organising in AI safety.
The flipside of the polarisation issue is the “false balance” issue, and that reference to smoking by TsviBT seems to be what this discussion is pointing at.
Admittedly, most of the reason why we are able to solve climate change easily while polarization happened is because it turned out to be the case that the problem was far easier to solve than feared (if we don’t care about animal welfare much, which is the case for ~all humans) without much government intervention.
I actually think this has a reasonable likelihood of happening, but conditional on no alignment solution that’s cheap enough to be adopted without large government support, if it’s doable at all, then polarization matters far more here, so it’s actually a useful case study for worlds where alignment is hard.
The climate change issue didn’t become polarized in other countries, and that’s good. It did get polarized here, and that’s bad. It has roadblocked even having discussions about solutions behind discussing the increasingly ridulous—but also increasingly prevelant—“question” of whether human-caused climate change is even real. People in the US questioned the reality of anthropogenic climate change MORE even as the evidence for it mounted—because it had become polarized, so was more about identity than facts and logic. See my AI scares and changing public beliefs for one graph of this maddening degredation of clarity.
So why create polarization on this issue?
The false balance issue is separate. One might suppose that creating polarization leads to false balance arguments, because then there are two sides so to be fair we should balance both of them. If there are just a range of opinions, false blance is less easy to argue for.
I don’t know what you mean by “the central claim” here.
I also don’t want to derail to actually discussing climate change; I just used it as one example in which polarization was pretty clearly really bad for solving a problem.
Sorry, this was perhaps unfair of me to pick on you for making the same sort of freehand argument that many have done, maybe I should write a top-level post about it.
To clarify—the idea that “climate change is not being solved because of polarisation” and “AI safety would suffer from being like climate action [due to the previous]” are twin claims that are not obvious. These arguments seem surface-level reasonable by hinging on a lot of internal American politics that I don’t think engages with the breadth of drivers of climate action. To some extent these arguments betray the lip service that AI safety is an international movement because they seek to explain the solution of an international problem solely within the framework of US politics. I also feel the polarisation of climate change is itself sensationalised.
But I think what you’ve said here is more interesting:
One might suppose that creating polarization leads to false balance arguments, because then there are two sides so to be fair we should balance both of them. If there are just a range of opinions, false blance is less easy to argue for.
It seems like you believe that the opposite of polarisation is plurality (all arguments seen as equally valid), whereas I would see the opposite of polarisation as consensus (one argument is seen as valid). This is in contrast to polarisation (different groups see different arguments as valid). Valid here being more like “respectable” rather than “100% accurate”. But indeed, it’s not obvious to me that the chain of causality is polarisation → desire for false balance, rather than desire for false balance → polarisation. (Also handwavey notion to the idea that this desire for false balance comes from conflicting goals a la conflict theory).
So it seems important to not help it move from being a discussion to a fight.
It seems like part of the practical implication of whatever you mean by this is to say:
Calling people kind of stupid for holding the position they do (which Tegmark’s framing definitely does)
Like, Tegmark’s post is pretty neutral, unless I’m missing something. So it sounds like you’re saying to not describe there being two camps at all. Is that roughly what you’re saying? I’m saying that in your abstract analysis of the situation, you should stop preventing yourself from understanding that there are two camps.
I’m just repeating Raemon’s sentiment and elaborating on some reasons to be so concerned with this. I agree with him that just not framing with the title “which side are you on” seems to have much the same upside and much less polarization downside.
The fact that there are people advocating for two incompatible strategies does not mean that there are two groups in other important senses. One could look at it, and I do, as a bunch of confused humans in a complex domain, none of whom have a very good grip on the real situation, and who fall on different sides of this policy issue, but could be persuaded to change their minds on it.
The title “Which side of the AI safety community are you in?” reifies the existence of two groups at odds, with some sort of group identity, and it doesn’t seem to be much benefit to making the call for signatures that way.
So yes, I’m objecting to using the term two groups at all, let alone in the title and as the central theme. Motivating people by stirring up resentment against an outgroup is a strategy as old as time. It works in the short term. But it has big long-term costs: now you have a conflict between groups instead of a bunch of people with a variety of opinions.
Suppose someone works for Anthropic, accords with the value placed on empiricism by their Core Views on AI Safety (March 2023) and gives any weight to the idea we are in the pessimistic scenario from that document.
I think they can reasonably sign the statement yet not want to assign themselves exclusively to either camp.
I pitched my tent as a Pause AI member and I guess camp B has formed nearby. But I also have empathy for the alternate version of me who judges the trade-offs differently and has ended up as above, with a camp A zipcode.
The A/B framing has value, but I strongly want to cooperate with that person and not sit in separate camps.
I felt confused at first when you said that this framing is leaning into polarization. I thought “I don’t see any clear red-tribe / blue-tribe affiliations here.”
Then I remembered that polarization doesn’t mean tying this issue to existing big coalitions (a la Hanson’s Policy Tug-O-War), but simply that it is causing people to factionalize and create an conflict and divide between them.
I think it seems to me like Max has correctly pointed out a significant crux about policy preferences between people who care about AI existential risk, and it also seems to me worth polling people and finding out who thinks what.
It does seem to me that the post is attempting to cause some factionalization here. I am interested in hearing about whether this is a good or bad faction to exist (relative to other divides) rather than simply saying that division is costly (which it is). I am interested in some argument about whether this is worth it / this faction is a real one.
Or perhaps you/others think it should ~never be actively pushed for in the way Max does in this post (or perhaps not in this way on a place with high standards for discourse like LW).
I think it is sometimes correct to specifically encourage factionalization, but I consider it bad form to do it on LessWrong, especially without being explicitly self-aware about it. (i.e should come with an acknowledging that you are spending down the epistemic commons and why you think it is worth it).
(Where, to be clear, it’s fine/good to say “hey guys I think there is a major, common disagreement here that is important to think about, and take actions based on.” The thing I’m objecting to is the title being “which side are you on?” and encouraging you to think in terms of sides, rather than a specific belief that you try to keep your identity small about.)
(like, even specifically resolving the lack-of-nuance this post complains about, requires distinguishing between “never build ASI” and “don’t build ASI until it can be done safely”, which isn’t covered in the Two Sides)
Camp B) “Don’t race to superintelligence”: People in this group typically argue that “racing to superintelligence is bad because of Y”. Here Y is typically some combination of “uncontrollable”, “1984”, “disempowerment” and “extinction”.
The main split is about whether racing in the current regime is desirable, so both “never build ASI” and “don’t build ASI until it can be done safely” fall within the scope of camp B. Call these two subcamps B1 and B2. I think B1 and B2 give the same prescriptions within the actionable timeframe.
don’t build ASI until it can be done safely > build ASI whenever but try to make it safe > never build ASI
Those people might give different prescriptions to the “never build ASI” people, like not endorsing actions that would tank the probability of ASI ever getting built. (Although in practice I think they probably mostly make the same prescriptions at the moment.)
In practice, bans can be lifted, so “never” is never going to become an unassailable law of the universe. And right now, it seems misguided to quibble over “Pause for 5, 10, 20 years”, and “Stop for good”, given the urgency of the extinction threat we are currently facing. If we’re going to survive the next decade with any degree of certainty, we need an alliance between B1 and B2, and I’m happy for one to exist.
I agree that some people have this preference ordering, but I don’t know of any difference in specific actionable recommendations that would be given by “don’t until safely” and “don’t ever” camps.
On this point specifically, those two groups are currently allied, though they don’t always recognize it. If sufficiently-safe alignment is found to be impossible or humanity decides to never build ASI, there would stop being any difference between the two groups.
This is well-encapsulated by the differences between Stop AI and PauseAI. At least from PauseAI’s perspective, both orgs are currently on exactly the same team.
Yes, my comment was meant to address the “never build ASI” and “don’t build ASI until it can be done safely” distinction, which Raemon was pointing out does not map onto Camp A and Camp B. All of ControlAI, PauseAI, and Stop AI are firmly in Camp B, but have different opinions about what to do once a moratorium is achieved.
One thing I meant to point toward was that unless we first coordinate to get that moratorium, the rest is a moot point.
I think there is some way that the conversation needs to advance, and I think this is roughly carving at some real joints and it’s important that people are tracking the distinction.
But
a) I’m generally worried about reifying the groups more into existence (as opposed to trying to steer towards a world where people can have more nuanced views). This is tricky, there are tradeoffs and I’m not sure how to handle this. But...
b) this post title and framing particular is super leaning into the polarization and I wish it did something different.
I don’t like polarization as such, but I also don’t like all of my loved ones being killed. I see this post and the open statement as dissolving a conflationary alliance that groups people who want to (at least temporarily) prevent the creation of superintelligence with people who don’t want to do that. Those two groups of people are trying to do very different things that I expect will have very different outcomes.
I don’t think the people in Camp A are immoral people just for holding that position[1], but I do think it is necessary to communicate: “If we do thing A, we will die. You must stop trying to do thing A, because that will kill everyone. Thing B will not kill everyone. These are not the same thing.”
In general, to actually get the things that you want in the world, sometimes you have to fight very hard for them, even against other people. Sometimes you have to optimize for convincing people. Sometimes you have to shame people. The norms of discourse that are comfortable for me and elevate truth-seeking and that make LessWrong a wonderful place are not always the same patterns as those that are most likely to cause us and our families to still be alive in the near future.
Though I have encountered some people in the AI Safety community who are happy to unnecessarily subject others to extreme risks without their consent after a naive utilitarian calculus on their behalf, which I do consider grossly immoral.
Sure—but are they in this case different? Is the polarization framing worth it here? I don’t think so, because polarization has large downsides.
You can argue against the “race to AGI” position without making the division into camps worse. Calling people kind of stupid for holding the position they do (which Tegmark’s framing definitely does) is a way to get people to dig in, double down, and fight you instead of quietly shifting their views based on logic.
I agree with him that position A is illogical; I think any reasonable estimate of risks demands more caution than the position he lays out. And I think that many people actually hold that position. But many have more nuanced views, which could firm up into a group identity and otherwise fairly risk-aware people putting effort into arguing against strong risk arguments based on defensive emotional responses.
Fine, and also I’m not saying what to do about it (shame or polarize or whatever), but PRIOR to that, we have to STOP PRETENDING IT’S JUST A VIEW. It’s a conflictual stance that they are taking. It’s like saying that the statisticians arguing against “smoking causes cancer” “have a nuanced view”.
I’m not pretending it’s just a view. The immense importance of this issue is another reason to avoid polarization. Look at how the climate change issue worked out with polarization involved.
The arguments for caution are very strong. Proponents of caution are at an advantage in a discussion. We’re also a minority, so we’re at a disadvantage in a fight. So it seems important to not help it move from being a discussion to a fight.
The climate change issue has pretty widespread international agreement and in most countries is considered a bipartisan issue. The capture of climate change by polarising forces has not really affected intervention outcomes (other problems of implementation are, imo, far greater).
I don’t want to derail the AI safety organising conversation, but I see this climate change comparison come up a lot. It strikes me as a pretty low-quality argument and it’s not clear a) whether the central claim is even true and b) whether it is transferable to organising in AI safety.
The flipside of the polarisation issue is the “false balance” issue, and that reference to smoking by TsviBT seems to be what this discussion is pointing at.
Admittedly, most of the reason why we are able to solve climate change easily while polarization happened is because it turned out to be the case that the problem was far easier to solve than feared (if we don’t care about animal welfare much, which is the case for ~all humans) without much government intervention.
I actually think this has a reasonable likelihood of happening, but conditional on no alignment solution that’s cheap enough to be adopted without large government support, if it’s doable at all, then polarization matters far more here, so it’s actually a useful case study for worlds where alignment is hard.
I don’t get it?
The climate change issue didn’t become polarized in other countries, and that’s good. It did get polarized here, and that’s bad. It has roadblocked even having discussions about solutions behind discussing the increasingly ridulous—but also increasingly prevelant—“question” of whether human-caused climate change is even real. People in the US questioned the reality of anthropogenic climate change MORE even as the evidence for it mounted—because it had become polarized, so was more about identity than facts and logic. See my AI scares and changing public beliefs for one graph of this maddening degredation of clarity.
So why create polarization on this issue?
The false balance issue is separate. One might suppose that creating polarization leads to false balance arguments, because then there are two sides so to be fair we should balance both of them. If there are just a range of opinions, false blance is less easy to argue for.
I don’t know what you mean by “the central claim” here.
I also don’t want to derail to actually discussing climate change; I just used it as one example in which polarization was pretty clearly really bad for solving a problem.
Sorry, this was perhaps unfair of me to pick on you for making the same sort of freehand argument that many have done, maybe I should write a top-level post about it.
To clarify—the idea that “climate change is not being solved because of polarisation” and “AI safety would suffer from being like climate action [due to the previous]” are twin claims that are not obvious. These arguments seem surface-level reasonable by hinging on a lot of internal American politics that I don’t think engages with the breadth of drivers of climate action. To some extent these arguments betray the lip service that AI safety is an international movement because they seek to explain the solution of an international problem solely within the framework of US politics. I also feel the polarisation of climate change is itself sensationalised.
But I think what you’ve said here is more interesting:
It seems like you believe that the opposite of polarisation is plurality (all arguments seen as equally valid), whereas I would see the opposite of polarisation as consensus (one argument is seen as valid). This is in contrast to polarisation (different groups see different arguments as valid). Valid here being more like “respectable” rather than “100% accurate”. But indeed, it’s not obvious to me that the chain of causality is polarisation → desire for false balance, rather than desire for false balance → polarisation. (Also handwavey notion to the idea that this desire for false balance comes from conflicting goals a la conflict theory).
It seems like part of the practical implication of whatever you mean by this is to say:
Like, Tegmark’s post is pretty neutral, unless I’m missing something. So it sounds like you’re saying to not describe there being two camps at all. Is that roughly what you’re saying? I’m saying that in your abstract analysis of the situation, you should stop preventing yourself from understanding that there are two camps.
I’m just repeating Raemon’s sentiment and elaborating on some reasons to be so concerned with this. I agree with him that just not framing with the title “which side are you on” seems to have much the same upside and much less polarization downside.
The fact that there are people advocating for two incompatible strategies does not mean that there are two groups in other important senses. One could look at it, and I do, as a bunch of confused humans in a complex domain, none of whom have a very good grip on the real situation, and who fall on different sides of this policy issue, but could be persuaded to change their minds on it.
The title “Which side of the AI safety community are you in?” reifies the existence of two groups at odds, with some sort of group identity, and it doesn’t seem to be much benefit to making the call for signatures that way.
So yes, I’m objecting to using the term two groups at all, let alone in the title and as the central theme. Motivating people by stirring up resentment against an outgroup is a strategy as old as time. It works in the short term. But it has big long-term costs: now you have a conflict between groups instead of a bunch of people with a variety of opinions.
Suppose someone works for Anthropic, accords with the value placed on empiricism by their Core Views on AI Safety (March 2023) and gives any weight to the idea we are in the pessimistic scenario from that document.
I think they can reasonably sign the statement yet not want to assign themselves exclusively to either camp.
I pitched my tent as a Pause AI member and I guess camp B has formed nearby. But I also have empathy for the alternate version of me who judges the trade-offs differently and has ended up as above, with a camp A zipcode.
The A/B framing has value, but I strongly want to cooperate with that person and not sit in separate camps.
I felt confused at first when you said that this framing is leaning into polarization. I thought “I don’t see any clear red-tribe / blue-tribe affiliations here.”
Then I remembered that polarization doesn’t mean tying this issue to existing big coalitions (a la Hanson’s Policy Tug-O-War), but simply that it is causing people to factionalize and create an conflict and divide between them.
I think it seems to me like Max has correctly pointed out a significant crux about policy preferences between people who care about AI existential risk, and it also seems to me worth polling people and finding out who thinks what.
It does seem to me that the post is attempting to cause some factionalization here. I am interested in hearing about whether this is a good or bad faction to exist (relative to other divides) rather than simply saying that division is costly (which it is). I am interested in some argument about whether this is worth it / this faction is a real one.
Or perhaps you/others think it should ~never be actively pushed for in the way Max does in this post (or perhaps not in this way on a place with high standards for discourse like LW).
I think it is sometimes correct to specifically encourage factionalization, but I consider it bad form to do it on LessWrong, especially without being explicitly self-aware about it. (i.e should come with an acknowledging that you are spending down the epistemic commons and why you think it is worth it).
(Where, to be clear, it’s fine/good to say “hey guys I think there is a major, common disagreement here that is important to think about, and take actions based on.” The thing I’m objecting to is the title being “which side are you on?” and encouraging you to think in terms of sides, rather than a specific belief that you try to keep your identity small about.)
(like, even specifically resolving the lack-of-nuance this post complains about, requires distinguishing between “never build ASI” and “don’t build ASI until it can be done safely”, which isn’t covered in the Two Sides)
The main split is about whether racing in the current regime is desirable, so both “never build ASI” and “don’t build ASI until it can be done safely” fall within the scope of camp B. Call these two subcamps B1 and B2. I think B1 and B2 give the same prescriptions within the actionable timeframe.
Some people likely think
Those people might give different prescriptions to the “never build ASI” people, like not endorsing actions that would tank the probability of ASI ever getting built. (Although in practice I think they probably mostly make the same prescriptions at the moment.)
In practice, bans can be lifted, so “never” is never going to become an unassailable law of the universe. And right now, it seems misguided to quibble over “Pause for 5, 10, 20 years”, and “Stop for good”, given the urgency of the extinction threat we are currently facing. If we’re going to survive the next decade with any degree of certainty, we need an alliance between B1 and B2, and I’m happy for one to exist.
I agree that some people have this preference ordering, but I don’t know of any difference in specific actionable recommendations that would be given by “don’t until safely” and “don’t ever” camps.
On this point specifically, those two groups are currently allied, though they don’t always recognize it. If sufficiently-safe alignment is found to be impossible or humanity decides to never build ASI, there would stop being any difference between the two groups.
This is well-encapsulated by the differences between Stop AI and PauseAI. At least from PauseAI’s perspective, both orgs are currently on exactly the same team.
Pause AI is clearly a central member of Camp B? And Holly signed the superintelligence petition.
Yes, my comment was meant to address the “never build ASI” and “don’t build ASI until it can be done safely” distinction, which Raemon was pointing out does not map onto Camp A and Camp B. All of ControlAI, PauseAI, and Stop AI are firmly in Camp B, but have different opinions about what to do once a moratorium is achieved.
One thing I meant to point toward was that unless we first coordinate to get that moratorium, the rest is a moot point.
Parenthetically, I do not yet know of anyone in the “never build ASI” camp and would be interested in reading or listening to such a person.
Shared without comment: https://www.stopai.info/
Émile Torres would be the most well-known person in that camp.
I just posted my attempt at combatting polarization a bit.