You’re painting a very binary, “us vs. them” picture of the option space here. There are many policy positions one might take between the extremes “datacenter moratorium” and “cheering the leading labs”; rejecting the idea of a moratorium does not mean you want the labs to be supercharged.
For some examples of such policies, see e.g. the Radical Optionality suggestions.
Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It’s not surprising to me that we’d see GPT2 level reasoning like “if you don’t vocally support a datacenter ban, you don’t believe in xrisk”, which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual’s brains running the Keynesian beauty contest, sees a low dimensional subspace where there’s a success policy vertex at the extrema “ban it all” and the consensus process is to push towards that with lesser policies incrementally.
So then someone says, for example, “banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all”, and it sounds like not wanting to hike through the intermediate policies to get the one you want.
I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.
I think the simple belief “it might kill everyone ⇒ shut it down” is a pretty good belief, and people rightly resist attempts to move them off it, and they are rightly skeptical of people who believe the first but not the second.
The crowd effect has more to do with the outgroup and less to do with matching what one’s ingroup thinks. If a stranger say “x-risk ⇒ build data centers”, they are not showing ingroup harmony, so they get evaluated for outgroup, and the answer is something like “I can easily see Sam and Dario lying about believing in x-risk as part of building hype, this stranger is a tech bro.” I wish those two were not the most visible examples of people who believe in x-risk.
I was thinking of nate soares or so in my previous comment—people who say things like “sure, whatever, don’t build datacenters, but that won’t really help very much”. If you’re really just thinking of sam/dario types, then yeah agreed that their positions are barely consistent in the terrain. I do think if they can end up right, then everything will be fine. I am as worried as I am because I think their chance of ending up right—“build the safe superintelligence first”—is extremely slim.
You’re painting a very binary, “us vs. them” picture of the option space here. There are many policy positions one might take between the extremes “datacenter moratorium” and “cheering the leading labs”; rejecting the idea of a moratorium does not mean you want the labs to be supercharged.
For some examples of such policies, see e.g. the Radical Optionality suggestions.
Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It’s not surprising to me that we’d see GPT2 level reasoning like “if you don’t vocally support a datacenter ban, you don’t believe in xrisk”, which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual’s brains running the Keynesian beauty contest, sees a low dimensional subspace where there’s a success policy vertex at the extrema “ban it all” and the consensus process is to push towards that with lesser policies incrementally.
So then someone says, for example, “banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all”, and it sounds like not wanting to hike through the intermediate policies to get the one you want.
I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.
I think the simple belief “it might kill everyone ⇒ shut it down” is a pretty good belief, and people rightly resist attempts to move them off it, and they are rightly skeptical of people who believe the first but not the second.
The crowd effect has more to do with the outgroup and less to do with matching what one’s ingroup thinks. If a stranger say “x-risk ⇒ build data centers”, they are not showing ingroup harmony, so they get evaluated for outgroup, and the answer is something like “I can easily see Sam and Dario lying about believing in x-risk as part of building hype, this stranger is a tech bro.” I wish those two were not the most visible examples of people who believe in x-risk.
I was thinking of nate soares or so in my previous comment—people who say things like “sure, whatever, don’t build datacenters, but that won’t really help very much”. If you’re really just thinking of sam/dario types, then yeah agreed that their positions are barely consistent in the terrain. I do think if they can end up right, then everything will be fine. I am as worried as I am because I think their chance of ending up right—“build the safe superintelligence first”—is extremely slim.