I noticed @Eliezer Yudkowskydistancing himself from Situational Awareness LP, without mentioning the connection via @CarlShulman who used to be a MIRI employee, and also that SALP just made a $500 million investment in a chip manufacturing startup, suggesting that Carl seems to be even more comfortable with accelerating AI development these days. (Unlike SALP’s investments in Anthropic and various public companies, which can be somewhat plausibly explained as safety motivated or not counterfactually meaningful, it seems hard to explain away the acceleration effect of investing in a pure hardware startup.)
To be more explicit about my implied criticism here, you’ve repeatedly drawn a line between “my people” (which I interpret as MIRI/rationalists/LWers/people who have absorbed the Sequences) and EAs, then criticizing EAs as a group, like in this latest tweet “EA seemingly continues its long-held tradition of doing the worst possible thing about AI.”
But as my upthread comments pointed out, Luke and Carl were prominent MIRI employees, rationalists, and LWers, before they became EAs. So how are they not “your people”? Or does someone automatically become not your people as soon as they become EAs? I find it hard to draw some kind of useful lesson or message from this thread of your thinking/comms (instead of interpreting it “uncharitably” as tribalism or avoidance of blame/responsibility).
FWIW, Carl’s and Luke’s epistemology (and to some degree moral intuitions) do just seem very different from Eliezer’s and Nate’s and mine (and this seemed true more than a decade ago to me). I have some trouble going into details because it feels like too much detailed gossip that’s semi-private, but I think there is a real meaningful distinction here.
(As an example, my career in the whole community started with Luke telling me in the MIRI offices in 2014 that people should stop doing rationality community building and start doing EA community building, which is less cringe and higher prestige in a way that will gain more traction among the competent and powerful. This was pretty consequential for me joining CEA less than a year later. I consider this among the worst advice I’ve gotten in terms of consequences on my life and career.)
1. Out-racing China is the biggest focus. We must publicly declare a Manhattan-style Project on superintelligence to beat China. When talking to the government, use the “beat China” framing.
2. Alignment of superintelligence is easy/solved. If it’s not solved, we solve it by telling AI to solve AI alignment for us. We don’t need to struggle thinking about that point any further; we can just make a mental note to tell the AIs at some point in the future to solve the problem for us.
3. AI stocks will soon allow them to buy literal galaxies, so the sooner we get ASI, the sooner they can get their own galaxies.
Based on these ideas, investing in new advanced lithography machines (AI chip making machines) makes perfect sense. Building more chips means out-racing China, and we just solve alignment by using a couple of these chips. And the sooner we get to ASI, the sooner they get to their galaxies.
I noticed @Eliezer Yudkowsky distancing himself from Situational Awareness LP, without mentioning the connection via @CarlShulman who used to be a MIRI employee, and also that SALP just made a $500 million investment in a chip manufacturing startup, suggesting that Carl seems to be even more comfortable with accelerating AI development these days. (Unlike SALP’s investments in Anthropic and various public companies, which can be somewhat plausibly explained as safety motivated or not counterfactually meaningful, it seems hard to explain away the acceleration effect of investing in a pure hardware startup.)
Thank you for the pointer; I have now tweeted about this and made the uncertain guess you did not particularly want a “H/T Wei Dai” about it.
To be more explicit about my implied criticism here, you’ve repeatedly drawn a line between “my people” (which I interpret as MIRI/rationalists/LWers/people who have absorbed the Sequences) and EAs, then criticizing EAs as a group, like in this latest tweet “EA seemingly continues its long-held tradition of doing the worst possible thing about AI.”
But as my upthread comments pointed out, Luke and Carl were prominent MIRI employees, rationalists, and LWers, before they became EAs. So how are they not “your people”? Or does someone automatically become not your people as soon as they become EAs? I find it hard to draw some kind of useful lesson or message from this thread of your thinking/comms (instead of interpreting it “uncharitably” as tribalism or avoidance of blame/responsibility).
FWIW, Carl’s and Luke’s epistemology (and to some degree moral intuitions) do just seem very different from Eliezer’s and Nate’s and mine (and this seemed true more than a decade ago to me). I have some trouble going into details because it feels like too much detailed gossip that’s semi-private, but I think there is a real meaningful distinction here.
(As an example, my career in the whole community started with Luke telling me in the MIRI offices in 2014 that people should stop doing rationality community building and start doing EA community building, which is less cringe and higher prestige in a way that will gain more traction among the competent and powerful. This was pretty consequential for me joining CEA less than a year later. I consider this among the worst advice I’ve gotten in terms of consequences on my life and career.)
From what I have seen, Leopold helped popularize these ideas in the AI safety discourse:
1. Out-racing China is the biggest focus. We must publicly declare a Manhattan-style Project on superintelligence to beat China. When talking to the government, use the “beat China” framing.
2. Alignment of superintelligence is easy/solved. If it’s not solved, we solve it by telling AI to solve AI alignment for us. We don’t need to struggle thinking about that point any further; we can just make a mental note to tell the AIs at some point in the future to solve the problem for us.
3. AI stocks will soon allow them to buy literal galaxies, so the sooner we get ASI, the sooner they can get their own galaxies.
Based on these ideas, investing in new advanced lithography machines (AI chip making machines) makes perfect sense. Building more chips means out-racing China, and we just solve alignment by using a couple of these chips. And the sooner we get to ASI, the sooner they get to their galaxies.