People who are attempting to cause serious harm need to be stopped. If someone is currently attempting murder a mad scientist is performing life-threatening experiments on people without their consent, it’s not reasonable to look for mutually beneficial arrangements with them, they need to be restrained and put in prison. I’m not open to peaceful coexistence with people who insist on building something that will likely destroy the human race. No compromise is possible, they simply can’t be allowed to build it. If they won’t stop voluntarily, they absolutely are enemies.
It seems to me that you’re making a bucket error between “someone trying to murder is on the other side of a conflict”, and “someone trying to murder is going to persistently keep trying to murder even if they obtain the things they say they want right now, because they at-least-somewhat-terminally value murder”. Perhaps there are ways to say things in language that is conditionally sedate and does not attribute the behavior you want to stop to an identity feature that the accused person is likely to internalize? For example, if someone is told “by participating in [x common thing], you are a murderer”, they seem more likely to consider themselves morally licensed to do other things that are called murder by angry people online. The argument isn’t “they’re not doing something that should be stopped”, which is what I see you as responding to; it’s “try not to write to their identity slot or to others’ identity slot for that person, when accusing them of bad behavior”.
“Mass manslaughterer” seems more accurate than “mass murderer” anyway, and might be lower on the scale OP describes.
Pushing toward ASI isn’t actually a common thing, only a tiny fraction of people are doing that. I think it’s unlikely, but if my words cause someone to quit pushing toward ASI but feel morally licensed to do other bad things, I think I’d still consider that a win, since pushing toward ASI is one of the worst things a person can do. I think there are people out there who at-least-somewhat-terminally value murder, but are prevented from committing murder by moral disapprobation and threat of punishment, so it’s important not to push those things outside the Overton window for fear of causing bad vibes.
I’m more worried about the case where your words not only don’t stop them from pushing towards ASI, but make them feel that in their pursuit of ASI, since they are now intended by you to consider themselves a bad person for doing it, they should think of themselves as being intentionally evil, and take other intentionally evil actions; or, someone else who would have tried to get them to stop by talking to them, now thinks of them as impossible to talk to, and treats them as beyond reach of communication and request. There’s a space between “say they’re doing a bad thing” and “say they’re inherently a kind of person who is inclined to do bad things”, or “say they’re impossible to pressure via ordinary means and must be pressured via unusual means”. I am not asking you to say what they’re doing is fine, and I would understand if the split that those-who-agree-with-OP are asking you to make wasn’t a split you were previously treating as notable.
Also, for the record, I’d volunteer my time to talk with anyone who is currently doing capabilities research at an AI research group, or who is seriously considering doing that, and try to explain why they shouldn’t but in a way that is kind and open and understanding. (I don’t think I have a legible track record of doing this, but I would unaccountably claim that there’s a significant chance I’d be good at this for a substantial subset of such people.)
Also, for the record, I’d volunteer my time to talk with anyone who is currently doing capabilities research at an AI research group, or who is seriously considering doing that, and try to explain why they shouldn’t but in a way that is kind and open and understanding. (I don’t think I have a legible track record of doing this, but I would unaccountably claim that there’s a significant chance I’d be good at this for a substantial subset of such people.)
There’s this one Upton Sinclair quote I think about a lot in this context. I imagine you’ve seen it?
Chewing it over more, I think you may have neglected to consider Newcomblike self-deception as a possible factor in Sinclair’s razor. It’s not necessary for the person to be lying about what they believe, or for them to have consciously convinced themselves of that lie. They can just have a big convenient cognitive blind spot.
Well, I meant that to be included under “has really convinced themselves”, where you’re proposing that they could have convinced themselves unconsciously. (Which I agree happens, via a bunch of little ugh fields and piecemeal distorted-world construction.)
Feel free to make an edit to clarify though, it’s a wiki!
I’ve always disliked the idea of people who assess risk differently being called “evil.” There’s a big difference between “I want to kill everyone, so I’m creating technology X” and “I disagree with the risk assessment of this technology and want to make the world a better place, so I’m creating it.”
I think the lesswrong community has become too radicalized in considering itself so epistemologically right that it calls people evil who simply disagree with them about the probability of p(doom).
UPD: The problem with this radicalization of the community is also that it repels many AI researchers who do not consider themselves evil and are less inclined to change their point of view on the risks posed by people who consider them evil and are probably not averse to killing them for disagreeing with them.
I actually think there are more people like that on LessWrong; the disagreement score on both their comment and this post has been going up and down a lot, so I think there is high variance. That is, I think there is a genuinely divisive conflict on LessWrong as to whether radicalization to the degree of calling AI developers “murderers” is good thing. (My position: it’s clearly a hyperbolic and thus irresponsible and/or bad-faith way to use the term ‘murder’.)
I did not call anyone a murderer in this thread. I did ask about it a month ago but replies convinced me it wasn’t appropriate. Although I see how even using it as an analogy could cause confusion. I’ll edit my comment.
I personally strong-downvoted the parent comment. When I remove my strong-downvote from it, I discover that the combined rest of LessWrong voted that comment to a total of +18 karma and +2 agreement as of me writing this.
I do in fact consider it somewhat alarming that the combined rest of LessWrong on net agrees with this comment and wants to reward the author for posting it.
Why are you asking me to put my own desire not to be killed along with everyone I care about at the bottom of my priorities list, below protecting the feelings of AI researchers? That is an insane request and I’m obviously not willing to do it.
I’d answer differently to Forza: I wouldn’t ask you to move it on your priorities list, I’d ask you to recognize that properly understood, prioritizing it means you do care about the feelings of AI researchers, and that you are making a mistake to treat their behavior as opaque. I’d like you to be trying to get them to stop more competently, which I don’t think involves telling them to feel like murderers separately from convincing them of the problem, because humans have known mental immune responses to being told how to feel in ways that are not justified by evidence they can directly process. I don’t mean to request a general update about all of your behavior, but I think your comments since my last ones don’t show evidence of having understood why I replied to you the way I did, or why I believe that the common memetic anger-pattern you are exhibiting here is a dominated strategy.
also, I explicitly do not claim that there is no memetic response warranted, nor that you can’t be mad. I just want you to recognize that people who might, in the full accounting of things, in fact qualify as ending up becoming the cause of mass death down the line, are in fact currently justified in being unsure whether that’s the case, and so being verbally confident at them is not a move that would be expected to change their behavior. Your pattern seems like one whose nearest effective variant is mass-movement-building, and I do think there are forms of that which can be effective; I think those will be the most effective if they’re highly palatable to contributors at labs while also not sacrificing moral clarity or factual-justification-to-an-uninformed-mind. I think you’re currently trying only for moral clarity, and moral clarity that results in autoimmune responses is in my view an unambiguously dominated strategy.
I’m not talking to AI researchers. I don’t think there’s any world where every AI researcher can talked into quitting, there will always be people lining up around the block to screw over humanity for 8+-figure amounts of money. This is why I think they have to be stopped. Talking an individual researcher into quitting may be good for that individual’s conscience, but only lengthens timelines by the difference between that person’s competence and their replacement, which probably isn’t enough to be worth much.
I don’t see why movement-building would be more effective if it’s highly palatable to contributors at labs, it seems more like the opposite is true, but feel free to explain.
because you might be wrong, or people might genuinely not understand that you’re right? There are many people who believe that their lives are in danger of a lot of things: vaccines against covid, GMOs, climate change, etc. They also sincerely believe that they are right and can consider scientists evil because they endanger their lives and the lives of their family. But the problem is that if they start equating any people who disagree with them with the evil that deliberately wants to destroy them, it radicalizes them and promoting enmity. Is this a rational approach when every person, confident that technology N can kill him, starts openly hating the scientists who develop it?
This is drawing a false equivalence between AI risk and random crackpot beliefs, which is dishonest. You won’t find accomplished scientists saying vaccines against covid, GMOs, or climate change have a substantial likelihood of killing everyone. Also it’s not relevant whether someone who’s going to kill me is evil and I’m not talking about that, I want them to be stopped from killing me whether they’re evil or not, and whether they can be talked out of it or not.
I would rather say that you should stop them in whatever way is most effective. If a peaceful compromise gets them to stop, do that. If forcing them to stop works better, do that. There is nothing morally wrong with trying to stop someone who’s trying to kill you, but that doesn’t mean it’s your best strategy.
If someone is currently attempting murder, it’s not reasonable to look for mutually beneficial arrangements with them, they need to be restrained and put in prison.
This is locally true, but also I want to point out similar situations where the reasoning isn’t quite the same:
If you’re at war with a country, of course you’re behaving as though “they need to be restrained and put in prison” or otherwise met with force. But I think often diplomacy and occasionally finding mutually-beneficial treaties is a good thing to do as well.
Often, political leaders are pushing for and against different policies on topics like war, terrorism, healthcare, foreign aid and more. Changes in policies here definitely cause deaths, often in their thousands (e.g. reallocating money from current life-saving efforts into R&D can look like this on paper), but it’s critical to the health of a nation for people to be able to disagree and talk about these things and advocate for different positions without facing force or violence, even if you know that a certain idea will have horrendous consequences.
I’m not open to peaceful coexistence with people who insist on building something that will likely destroy the human race.
I myself would support laws imprisoning people selling out humanity and getting rich in the process (by building AGI via ML scaling), but I think violence is a really important civilizational schelling line and very dangerous to cross, and, while I admit I am still confused about the lines here, I am uncomfortable reading this. Your comment pattern matches to me with a pretty naive escalation across ethical lines, rather than a considered and mature one. You’re vaguely threatening violence almost reactively, rather than making any case that it would improve things or that this is one of the rare times where it’s okay to cross the line. This rhymes for me with a childish, impotent desire to have power over people who are doing terrible things, which is understandable, but doesn’t just any means of doing it.
I was vague because I don’t think it’s actually prudent to threaten anyone at this time, but I do think it’s important to defend the possibility of talking about it. Of course I know it’s possible to be counterproductively aggressive, but I guess I’m getting a little edgy because my sense is that almost all people reading LW err on the side of being way too conflict-avoidant. Being ready and willing to fight (including nonviolent resistance) can have advantages even if no fighting actually occurs, but it requires, among other things, being able to identify enemies.
People who are attempting to cause serious harm need to be stopped.
This is such a weird phrasing, because it attributes the need to the person it’s proposing to stop, instead of to the people who want them stopped.
Consider: If Julius Caesar is attempting to cause serious harm to the Gauls, it’s not Caesar who needs Caesar to be stopped; it’s the Gauls who need Caesar to be stopped.
People who are attempting to cause serious harm need to be stopped. If
someone is currently attempting murdera mad scientist is performing life-threatening experiments on people without their consent, it’s not reasonable to look for mutually beneficial arrangements with them, they need to be restrained and put in prison. I’m not open to peaceful coexistence with people who insist on building something that will likely destroy the human race. No compromise is possible, they simply can’t be allowed to build it. If they won’t stop voluntarily, they absolutely are enemies.It seems to me that you’re making a bucket error between “someone trying to murder is on the other side of a conflict”, and “someone trying to murder is going to persistently keep trying to murder even if they obtain the things they say they want right now, because they at-least-somewhat-terminally value murder”. Perhaps there are ways to say things in language that is conditionally sedate and does not attribute the behavior you want to stop to an identity feature that the accused person is likely to internalize? For example, if someone is told “by participating in [x common thing], you are a murderer”, they seem more likely to consider themselves morally licensed to do other things that are called murder by angry people online. The argument isn’t “they’re not doing something that should be stopped”, which is what I see you as responding to; it’s “try not to write to their identity slot or to others’ identity slot for that person, when accusing them of bad behavior”.
“Mass manslaughterer” seems more accurate than “mass murderer” anyway, and might be lower on the scale OP describes.
Pushing toward ASI isn’t actually a common thing, only a tiny fraction of people are doing that. I think it’s unlikely, but if my words cause someone to quit pushing toward ASI but feel morally licensed to do other bad things, I think I’d still consider that a win, since pushing toward ASI is one of the worst things a person can do. I think there are people out there who at-least-somewhat-terminally value murder, but are prevented from committing murder by moral disapprobation and threat of punishment, so it’s important not to push those things outside the Overton window for fear of causing bad vibes.
I’m more worried about the case where your words not only don’t stop them from pushing towards ASI, but make them feel that in their pursuit of ASI, since they are now intended by you to consider themselves a bad person for doing it, they should think of themselves as being intentionally evil, and take other intentionally evil actions; or, someone else who would have tried to get them to stop by talking to them, now thinks of them as impossible to talk to, and treats them as beyond reach of communication and request. There’s a space between “say they’re doing a bad thing” and “say they’re inherently a kind of person who is inclined to do bad things”, or “say they’re impossible to pressure via ordinary means and must be pressured via unusual means”. I am not asking you to say what they’re doing is fine, and I would understand if the split that those-who-agree-with-OP are asking you to make wasn’t a split you were previously treating as notable.
Agreed. Cf. https://www.lesswrong.com/posts/CYTwRZtrhHuYf7QYu/a-case-for-courage-when-speaking-of-ai-danger?commentId=pLH6dxnTrTz56BQYj
Also, for the record, I’d volunteer my time to talk with anyone who is currently doing capabilities research at an AI research group, or who is seriously considering doing that, and try to explain why they shouldn’t but in a way that is kind and open and understanding. (I don’t think I have a legible track record of doing this, but I would unaccountably claim that there’s a significant chance I’d be good at this for a substantial subset of such people.)
There’s this one Upton Sinclair quote I think about a lot in this context. I imagine you’ve seen it?
I wrote the wiki entry on it :) https://www.lesswrong.com/w/sinclair-s-razor
Chewing it over more, I think you may have neglected to consider Newcomblike self-deception as a possible factor in Sinclair’s razor. It’s not necessary for the person to be lying about what they believe, or for them to have consciously convinced themselves of that lie. They can just have a big convenient cognitive blind spot.
https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC/tsvibt-s-shortform?commentId=WsrmFhuysbmJ3xiPm
Well, I meant that to be included under “has really convinced themselves”, where you’re proposing that they could have convinced themselves unconsciously. (Which I agree happens, via a bunch of little ugh fields and piecemeal distorted-world construction.)
Feel free to make an edit to clarify though, it’s a wiki!
I’ve always disliked the idea of people who assess risk differently being called “evil.” There’s a big difference between “I want to kill everyone, so I’m creating technology X” and “I disagree with the risk assessment of this technology and want to make the world a better place, so I’m creating it.”
I think the lesswrong community has become too radicalized in considering itself so epistemologically right that it calls people evil who simply disagree with them about the probability of p(doom).
UPD: The problem with this radicalization of the community is also that it repels many AI researchers who do not consider themselves evil and are less inclined to change their point of view on the risks posed by people who consider them evil and are probably not averse to killing them for disagreeing with them.
You are responding to one person who received heavy disagreement in comments.
I actually think there are more people like that on LessWrong; the disagreement score on both their comment and this post has been going up and down a lot, so I think there is high variance. That is, I think there is a genuinely divisive conflict on LessWrong as to whether radicalization to the degree of calling AI developers “murderers” is good thing. (My position: it’s clearly a hyperbolic and thus irresponsible and/or bad-faith way to use the term ‘murder’.)
I did not call anyone a murderer in this thread. I did ask about it a month ago but replies convinced me it wasn’t appropriate. Although I see how even using it as an analogy could cause confusion. I’ll edit my comment.
I personally strong-downvoted the parent comment. When I remove my strong-downvote from it, I discover that the combined rest of LessWrong voted that comment to a total of +18 karma and +2 agreement as of me writing this.
I do in fact consider it somewhat alarming that the combined rest of LessWrong on net agrees with this comment and wants to reward the author for posting it.
Why are you asking me to put my own desire not to be killed along with everyone I care about at the bottom of my priorities list, below protecting the feelings of AI researchers? That is an insane request and I’m obviously not willing to do it.
I’d answer differently to Forza: I wouldn’t ask you to move it on your priorities list, I’d ask you to recognize that properly understood, prioritizing it means you do care about the feelings of AI researchers, and that you are making a mistake to treat their behavior as opaque. I’d like you to be trying to get them to stop more competently, which I don’t think involves telling them to feel like murderers separately from convincing them of the problem, because humans have known mental immune responses to being told how to feel in ways that are not justified by evidence they can directly process. I don’t mean to request a general update about all of your behavior, but I think your comments since my last ones don’t show evidence of having understood why I replied to you the way I did, or why I believe that the common memetic anger-pattern you are exhibiting here is a dominated strategy.
also, I explicitly do not claim that there is no memetic response warranted, nor that you can’t be mad. I just want you to recognize that people who might, in the full accounting of things, in fact qualify as ending up becoming the cause of mass death down the line, are in fact currently justified in being unsure whether that’s the case, and so being verbally confident at them is not a move that would be expected to change their behavior. Your pattern seems like one whose nearest effective variant is mass-movement-building, and I do think there are forms of that which can be effective; I think those will be the most effective if they’re highly palatable to contributors at labs while also not sacrificing moral clarity or factual-justification-to-an-uninformed-mind. I think you’re currently trying only for moral clarity, and moral clarity that results in autoimmune responses is in my view an unambiguously dominated strategy.
see also btfc’s comments on soft language
I’m not talking to AI researchers. I don’t think there’s any world where every AI researcher can talked into quitting, there will always be people lining up around the block to screw over humanity for 8+-figure amounts of money. This is why I think they have to be stopped. Talking an individual researcher into quitting may be good for that individual’s conscience, but only lengthens timelines by the difference between that person’s competence and their replacement, which probably isn’t enough to be worth much.
I don’t see why movement-building would be more effective if it’s highly palatable to contributors at labs, it seems more like the opposite is true, but feel free to explain.
because you might be wrong, or people might genuinely not understand that you’re right? There are many people who believe that their lives are in danger of a lot of things: vaccines against covid, GMOs, climate change, etc. They also sincerely believe that they are right and can consider scientists evil because they endanger their lives and the lives of their family. But the problem is that if they start equating any people who disagree with them with the evil that deliberately wants to destroy them, it radicalizes them and promoting enmity. Is this a rational approach when every person, confident that technology N can kill him, starts openly hating the scientists who develop it?
This is drawing a false equivalence between AI risk and random crackpot beliefs, which is dishonest. You won’t find accomplished scientists saying vaccines against covid, GMOs, or climate change have a substantial likelihood of killing everyone. Also it’s not relevant whether someone who’s going to kill me is evil and I’m not talking about that, I want them to be stopped from killing me whether they’re evil or not, and whether they can be talked out of it or not.
I would rather say that you should stop them in whatever way is most effective. If a peaceful compromise gets them to stop, do that. If forcing them to stop works better, do that. There is nothing morally wrong with trying to stop someone who’s trying to kill you, but that doesn’t mean it’s your best strategy.
Totally agree.
This is locally true, but also I want to point out similar situations where the reasoning isn’t quite the same:
If you’re at war with a country, of course you’re behaving as though “they need to be restrained and put in prison” or otherwise met with force. But I think often diplomacy and occasionally finding mutually-beneficial treaties is a good thing to do as well.
Often, political leaders are pushing for and against different policies on topics like war, terrorism, healthcare, foreign aid and more. Changes in policies here definitely cause deaths, often in their thousands (e.g. reallocating money from current life-saving efforts into R&D can look like this on paper), but it’s critical to the health of a nation for people to be able to disagree and talk about these things and advocate for different positions without facing force or violence, even if you know that a certain idea will have horrendous consequences.
I myself would support laws imprisoning people selling out humanity and getting rich in the process (by building AGI via ML scaling), but I think violence is a really important civilizational schelling line and very dangerous to cross, and, while I admit I am still confused about the lines here, I am uncomfortable reading this. Your comment pattern matches to me with a pretty naive escalation across ethical lines, rather than a considered and mature one. You’re vaguely threatening violence almost reactively, rather than making any case that it would improve things or that this is one of the rare times where it’s okay to cross the line. This rhymes for me with a childish, impotent desire to have power over people who are doing terrible things, which is understandable, but doesn’t just any means of doing it.
I was vague because I don’t think it’s actually prudent to threaten anyone at this time, but I do think it’s important to defend the possibility of talking about it. Of course I know it’s possible to be counterproductively aggressive, but I guess I’m getting a little edgy because my sense is that almost all people reading LW err on the side of being way too conflict-avoidant. Being ready and willing to fight (including nonviolent resistance) can have advantages even if no fighting actually occurs, but it requires, among other things, being able to identify enemies.
This is such a weird phrasing, because it attributes the need to the person it’s proposing to stop, instead of to the people who want them stopped.
Consider: If Julius Caesar is attempting to cause serious harm to the Gauls, it’s not Caesar who needs Caesar to be stopped; it’s the Gauls who need Caesar to be stopped.
I used the passive voice because identifying who does the stopping isn’t directly relevant to the topic of whether it’s good or bad to promote enmity.