Sam Altman and Dario Amodei are the public faces of AI safety and that’s poisoning the well
Normies just smell a rat whenever those two bring up AI safety. I think when the AI safety community gets a word in edge-wise (like you’re arguing with your friends or maybe a tweet goes viral), we usually say something like
“it could kill everyone. but China could also kill everyone, so now more than ever we have to supercharge OpenAI and Anthropic.”
and we just sound like we’re “one of them.”
I feel like I should be hearing “shut it down” about as often as I hear “broke containment” and that’s not really happening, instead it’s more like “let OpenAI cook.” So a number of people are pretty soured on AI safety because of this.
When have AI safety people said that OpenAI and Anthropic should be supercharged? Maybe you hang out with a very different set of AI safety people, because this has not been my experience at all.
A lot of AI safety people say things about Dario Amodei’s decision to move from a contract that forbids the military from allowing to use Claude to run disinformation campaigns to sway European elections to offering one that allows the military to do so as Dario standing up for his principles.
Given that both the first Trump and Biden administration ran their antivax disinformation campaign in the Philippines, thinking that no harm will come from allowing the present Trump administration to run Claude supported disinformation campaigns is bad.
Anthropic did suffer consequences for not rolling over on autonomous killing decisions and domestic surveillance but the decision was still giving up a good portion of the principles that are actually important.
I think this concretely comes up with “do you support a datacenter moratorium?” If you say no, you’ve taken a confusing stance with respect to the “it might kill everyone” stance.
I’ll say you have a point, I can revise like this: the public does sometimes get exposure to a genuine AIS x-risk argument, but they reject it because they mostly hear about x-risk from AI CEOs who conspicuously don’t say “shut it down.” (Which is not the same as “the AIS person said don’t shut it down.”)
And not everyone rejects it, and the Overton window is probably moving to include mass unemployment and x-risk.
You’re painting a very binary, “us vs. them” picture of the option space here. There are many policy positions one might take between the extremes “datacenter moratorium” and “cheering the leading labs”; rejecting the idea of a moratorium does not mean you want the labs to be supercharged.
For some examples of such policies, see e.g. the Radical Optionality suggestions.
Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It’s not surprising to me that we’d see GPT2 level reasoning like “if you don’t vocally support a datacenter ban, you don’t believe in xrisk”, which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual’s brains running the Keynesian beauty contest, sees a low dimensional subspace where there’s a success policy vertex at the extrema “ban it all” and the consensus process is to push towards that with lesser policies incrementally.
So then someone says, for example, “banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all”, and it sounds like not wanting to hike through the intermediate policies to get the one you want.
I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.
I think the simple belief “it might kill everyone ⇒ shut it down” is a pretty good belief, and people rightly resist attempts to move them off it, and they are rightly skeptical of people who believe the first but not the second.
The crowd effect has more to do with the outgroup and less to do with matching what one’s ingroup thinks. If a stranger say “x-risk ⇒ build data centers”, they are not showing ingroup harmony, so they get evaluated for outgroup, and the answer is something like “I can easily see Sam and Dario lying about believing in x-risk as part of building hype, this stranger is a tech bro.” I wish those two were not the most visible examples of people who believe in x-risk.
I was thinking of nate soares or so in my previous comment—people who say things like “sure, whatever, don’t build datacenters, but that won’t really help very much”. If you’re really just thinking of sam/dario types, then yeah agreed that their positions are barely consistent in the terrain. I do think if they can end up right, then everything will be fine. I am as worried as I am because I think their chance of ending up right—“build the safe superintelligence first”—is extremely slim.
Sam Altman and Dario Amodei are the public faces of AI safety and that’s poisoning the well
Normies just smell a rat whenever those two bring up AI safety. I think when the AI safety community gets a word in edge-wise (like you’re arguing with your friends or maybe a tweet goes viral), we usually say something like
“it could kill everyone. but China could also kill everyone, so now more than ever we have to supercharge OpenAI and Anthropic.”
and we just sound like we’re “one of them.”
I feel like I should be hearing “shut it down” about as often as I hear “broke containment” and that’s not really happening, instead it’s more like “let OpenAI cook.” So a number of people are pretty soured on AI safety because of this.
When have AI safety people said that OpenAI and Anthropic should be supercharged? Maybe you hang out with a very different set of AI safety people, because this has not been my experience at all.
Less_raichu seems to be saying that those two say that thing and that they’re the loudest spokespersons for ai safety.
A lot of AI safety people say things about Dario Amodei’s decision to move from a contract that forbids the military from allowing to use Claude to run disinformation campaigns to sway European elections to offering one that allows the military to do so as Dario standing up for his principles.
Given that both the first Trump and Biden administration ran their antivax disinformation campaign in the Philippines, thinking that no harm will come from allowing the present Trump administration to run Claude supported disinformation campaigns is bad.
Anthropic did suffer consequences for not rolling over on autonomous killing decisions and domestic surveillance but the decision was still giving up a good portion of the principles that are actually important.
I think this concretely comes up with “do you support a datacenter moratorium?” If you say no, you’ve taken a confusing stance with respect to the “it might kill everyone” stance.
I’ll say you have a point, I can revise like this: the public does sometimes get exposure to a genuine AIS x-risk argument, but they reject it because they mostly hear about x-risk from AI CEOs who conspicuously don’t say “shut it down.” (Which is not the same as “the AIS person said don’t shut it down.”)
And not everyone rejects it, and the Overton window is probably moving to include mass unemployment and x-risk.
You’re painting a very binary, “us vs. them” picture of the option space here. There are many policy positions one might take between the extremes “datacenter moratorium” and “cheering the leading labs”; rejecting the idea of a moratorium does not mean you want the labs to be supercharged.
For some examples of such policies, see e.g. the Radical Optionality suggestions.
Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It’s not surprising to me that we’d see GPT2 level reasoning like “if you don’t vocally support a datacenter ban, you don’t believe in xrisk”, which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual’s brains running the Keynesian beauty contest, sees a low dimensional subspace where there’s a success policy vertex at the extrema “ban it all” and the consensus process is to push towards that with lesser policies incrementally.
So then someone says, for example, “banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all”, and it sounds like not wanting to hike through the intermediate policies to get the one you want.
I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.
I think the simple belief “it might kill everyone ⇒ shut it down” is a pretty good belief, and people rightly resist attempts to move them off it, and they are rightly skeptical of people who believe the first but not the second.
The crowd effect has more to do with the outgroup and less to do with matching what one’s ingroup thinks. If a stranger say “x-risk ⇒ build data centers”, they are not showing ingroup harmony, so they get evaluated for outgroup, and the answer is something like “I can easily see Sam and Dario lying about believing in x-risk as part of building hype, this stranger is a tech bro.” I wish those two were not the most visible examples of people who believe in x-risk.
I was thinking of nate soares or so in my previous comment—people who say things like “sure, whatever, don’t build datacenters, but that won’t really help very much”. If you’re really just thinking of sam/dario types, then yeah agreed that their positions are barely consistent in the terrain. I do think if they can end up right, then everything will be fine. I am as worried as I am because I think their chance of ending up right—“build the safe superintelligence first”—is extremely slim.