This is a review of the reviews
This is a review of the reviews, a meta review if you will, but first a tangent. and then a history lesson. This felt boring and obvious and somewhat annoying to write, which apparently writers say is a good sign to write about the things you think are obvious. I felt like pointing towards a thing I was noticing, like 36 hours ago, which in internet speed means this is somewhat cached. Alas.
Previously, I rode a motorcycle. I rode it for about a year while working on semiconductors until I got a concussion, which slowed me down but did not update me to stop, until it eventually got stolen. The risk in dying from riding a motorcycle for a year is about 1 in 800 depending on the source.
Previously, I sailed across an ocean. I wanted to calibrate towards how dangerous it was. The forums said the probability of dying from a transatlantic crossing is like 1 in 10,000.
Currently, the people I know working on AI are far more well calibrated to the risk of AI than the general public, and even me, and almost all of them I know think there is more than a 5% chance of an AI catastrophe. That is a 1 in 20 chance, which feels recklessly high.
The thing I wanted to point to was the mental gymnastics I observed in peoples book reviews (if im feeling more contentious I might come back and link to some examples) and the way it made me both disappointed and almost not want to say anything.
I think it’s virtuous to note disagreements and it’s cool to note agreements, but it’s also even cooler and virtuous to avoid misleading people by not saying the trees are not there, when getting into the weeds, literally between the trees.
There are a bunch of people who are allegedly trying to change the world here. We all allegedly think lots of stuff is at stake. Building a coalition doesn’t look like suppressing disagreements, but it does look like building around the areas of agreement.
If you think there’s a 1 in 20 chance it could be so over, it feels to me the part where people are not doing the ‘yes the situation is insane’ even if that is immediately followed up with ‘im more hopeful than them tbc’ is weird.
On the first day of improv classes they teach you to say ‘yes, and’ instead of using ‘no’, which can kill the scene, when you don’t know how to respond, or to move things along. So, yes, and -
Now is time for the history lesson. The Shanghai Communiqué..
The Shanghai Communiqué was a joint statement issued by the U.S. and China in 1972, breaking a plus 20 year freeze with no diplomatic relations. The communiqué ended a long period of isolation between the two countries and paved the way towards a formal normalization seven years later.
I think some people critique it for pushing the Taiwan issue down the line, but I like to think about things in the context in which they existed. There was a threat great enough that countries had the incentive to coordinate together.
These negotiations were happening in the shadow of the cold war, and relations with China were also about counter balancing the USSR. The Sino-Soviet split created space for triangular diplomacy, and improved relations with China, could pressure the Soviets to cooperate on a different arms race.
I bring up the communiqué because I think it is cool. It acknowledged disagreements, even clearly laid them out, and used some well worded phrases to get it across the line. But the success of the negotiation is, in my read, attributed to the focus on the areas of agreement. Focusing on disagreement would have sunk it before it even started.
It worries me that people who allegedly care about the future going well, and are also at least 5% concerned that AI is not going to go well, are also squandering opportunities to help wake the world up to the dangers that they themselves are seeing, even if they see them slightly differently from the authors.
That said, I am slightly more hopeful than I was yesterday, and hope to feel further more hopeful in tomorrow.
This seems wise. The reception of the book in the community has been rather Why Our Kind Can’t Cooperate, as someone whom I forget linked. The addiction to hashing-out-object-level-correctness-on-every-point-of-factual-disagreement and insistence on “everything must be simulacrum level 0 all the time”… well, it’s not particularly conducive to getting things done in the real world.
I’m not suggesting we become propagandists, but I think pretty much every x-risk-worried Rat who disliked the book because e.g. the evolution analogy doesn’t work, they would have preferred a different flavor of sci-fi story, or the book should have been longer, or it should have been shorter, or it should have proposed my favorite secret plan for averting doom, or it should have contained draft legislation at the back… if they would endorse such a statement, I think that (metaphorically) there should be an all-caps disclaimer that reads something like “TO BE CLEAR AI IS STILL ON TRACK TO KILL EVERYONE YOU LOVE; YOU SHOULD BE ALARMED ABOUT THIS AND TELLING PEOPLE IN NO UNCERTAIN TERMS THAT YOU HAVE FAR, FAR MORE IN COMMON WITH YUDKOWSKY AND SOARES THAN YOU DO WITH THE LOBBYISTS OF META, WHO ABSENT COORDINATION BY PEOPLE ON HUMANITY’S SIDE ARE LIABLE TO WIN THIS FIGHT, SO COORDINATE WE MUST” every couple of paragraphs.
I don’t mean to say that the time for words and analysis is over. It isn’t. But the time for action has begun, and words are a form of action. That’s what’s missing, is the words-of-action. It’s a missing mood. Parable (which, yes, I have learned some people find really annoying):
A pale, frightened prisoner of war returns to the barracks, where he tells his friend: “Hey man, I heard the guards talking, and I think they’re gonna take us out, make us a dig a ditch, and then shoot us in the back. This will happen at dawn on Thursday.”
The friend snorts, “Why would they shoot us in the back? That’s incredibly stupid. Obviously they’ll shoot us in the head; it’s more reliable. And do they really need for us to dig a ditch first? I think they’ll just leave us to the jackals. Besides, the Thursday thing seems over-confident. Plans change around here, and it seems more logical for it to happen right before the new round of prisoners comes in, which is typically Saturday, so they could reasonably shoot us Friday. Are you sure you heard Thursday?”
The second prisoner is making some good points. He is also, obviously, off his rocker.
There are two steelmen I can think of here. One is “We must never abandon this relentless commitment to precise truth. All we say, whether to each other or to the outside world, must be thoroughly vetted for its precise truthfulness.” To which my reply is: how’s that been working out for us so far? Again, I don’t suggest we turn to outright lying like David Sacks, Perry Metzger, Sam Altman, and all the other rogues. But would it kill us to be the least bit strategic or rhetorical? Politics is the mind-killer, sure. But ASI is the planet-killer, and politics is the ASI-[possibility-thereof-]killer, so I am willing to let my mind take a few stray bullets.
The second is “No, the problems I have with the book are things that will critically undermine its rhetorical effectiveness. I know the heart of the median American voter, and she’s really gonna hate this evolution analogy.” To which I say, “This may be so. The confidence and negativity with which you have expressed this disagreement are wholly unwarranted.”
Let’s win, y’all. We can win without sacrificing style and integrity. It might require everyone to sacrifice a bit of personal pride, a bit of delight-in-one’s-own-cleverness. I’m not saying keep objections to yourself. I am saying, keep your eye on the fucking ball. The ball is not “being right,” the ball is survival.
I was pushing back on a similar attitude yesterday on twitter → LINK.
Basically, I’m in favor of people having nitpicky high-decoupling discussion on lesswrong, and meanwhile doing rah rah activism action PR stuff on twitter and bluesky and facebook and intelligence.org and pauseai.info and op-eds and basically the entire rest of the internet and world. Just one website of carve-out. I don’t think this is asking too much!
Yeah, I agree. The audience for this book isn’t LessWrong, but lots of people seem to be acting as if pushing back on LessWrong is a defection that will hurt the book’s prospects.
That’s fair!
I’m in the Twitter thread with Steve. I’ll just note that I don’t think it’s realistic to expect the world’s reaction to be more passionate and supportive than the LW community’s signaled reaction.
Why not? It seems extremely reasonable to have a place for persnickety internal-ish discussion, and other content somewhere else?
Are most of the persnickety internal-disagreers actually signalling that they intend to promote the book, or at least not downplay its thesis? I don’t think rationalists at large have a great track record of engaging the outside world on a unified front, or in leaving nuance aside when the nuance would stand in the way of the important parts of the communication. In other words, I don’t think the two types of content are on different platforms. I think it’s usually the same content on both.
In general, I’ve noticed that a lot of people think “scout mindset” means never having to pick up a (metaphorical) rifle. That’s a good way to have a precise model of how you’re going to die, without having any hand in preventing it. The most useful people in the world right now are scouts who are willing to act like soldiers from time to time.
One of the persnickety internal disagreers here. I have recommended IABIED to those of my acquaintances who I expect may read it. I don’t really have any other platform to shout about it from, but if I did, I would’ve certainly used it to promote the book, leaving all nitpicking out of it.
I, at least, do explicitly make a distinction between “a place for persnickety internal discussion” and “the public-facing platform”, and would behave differently between the two.
In principle it makes sense. But in reality right now, the only place where there’s a sizable MIRI-aligned community, is the community that’s entirely going the persnickety route. I’m open to different counterfactual comparisons, I’m just noting that compared to the world where there’s a sizable MIRI-aligned community that shows support for MIRI, this world is disappointing.
LessWrong is not an activist community, and should not become one. I think there are some promising arguments for trying to create activist spaces and communities (as well as some substantially valid warnings). I am currently kind of confused about how good it would be to create more of those spaces, but I think if it’s a good idea, people should not attempt to try to make LessWrong into one.
I don’t see “how you express yourself on a highly argumentative web forum” as limiting “how you express yourself at a launch party” or “how you express yourself on a popular podcast” other places.
But you just did propose sacrificing our integrity: specifically, the integrity of our relentless commitment to precise truth. It was two paragraphs ago. The text is right there. We can see it. Do you expect us not to notice?
To be clear, in this comment, I’m not even arguing that you’re wrong. Given the situation, maybe sacrificing the integrity of our relentless commitment to precise truth is exactly what’s needed!
But you can’t seriously expect people not to notice, right? You are including the costs of people noticing as part of your consequentialist decision calculus, right?
No, I just expressed myself badly. Thanks for keeping me honest. Let me try to rephrase—in response to any text, you can write ~arbitrarily many words in reply that lay out exactly where it was wrong. You can also write ~arbitrarily many words in reply that lay out where it was right. You can vary not only the quantity but the stridency/emphasis of these collections of words. (I’m only talking simulacrum-0 stuff here.) This is no canonical weighting of these!! You have to choose. The choice is not determined by your commitment to speaking truth. The choice is determined by priorities about how your words move others’ minds and move the world. Does that make more sense?
‘Speak only truth’ is underconstrained; we’ve allowed ourselves to add (charitably) ‘and speak all the truth that your fingers have the strength to type, particularly on topics about which there appears to be disagreement’ or (uncharitably) ‘and cultivate the aesthetic of a discerning, cantankerous, genius critic’ in order to get lower-dimensional solutions.
When constraints don’t eliminate all dimensions, I think you can reasonably have lexically ordered preferences. We’ve picked a good first priority (speak only truth), but have picked a counterproductive second priority ([however you want to describe it]). I claim our second priority should be something like “and accomplish your goals.” Where your goals, presumably, = survive.
OK, I am rereading what I wrote last night and I see that I really expressed myself badly. It really does sound like I said we shoudl sacrifice our commitment to precise truth. I’ll try again: what we should indeed sacrifice is our commitment to being anal-retentive about practices that we think associate with getting the precise truth, over and beyond saying true stuff and contradicting false stuff. where those practices include things like “never appearing to ‘rally round anything’ in a tribal fashion.” Or, at a 20degree angle from that: “doing rhetoric not with an aim toward an external goal, but orienting our rhetoric to be ostentatious in our lack of rhetoric, making all the trappings of our speech scream ‘this is a scrupulous, obsessive, nonpartisan autist for the truth.’” Does that make more sense? it’s the performative elements that get my goat. (And yes, there are performative elements, unavoidably! All speech has rhetoric because (metaphorically) “the semantic dimensions” are a subspace of speech-space, and speech-space is affine, so there’s no way to “set the non-semantic dimensions to zero.”)
This paragraph feels righter-to-me (oh, huh, you even ended up with the same word “ostentatious” as pointer that I did in my comment-1-minute-ago)
This is important enough that you should clarify in your own words. Raymond Arnold, as a moderator of lesswrong.com, is it in fact your position that “what we should indeed sacrifice is our commitment to being anal-retentive about practices that we think associate with getting the precise truth, over and beyond saying true stuff and contradicting false stuff”?
The word and actual connotations of anal-retentive are important to my sentence. (Also, I said “this feels righter-to-me” not “this is right” and I definitely did not make an explicit defense of exactly this wording as aspirational policy)
We absolutely should have more practices that drive at the precise truth than saying true stuff and contradicting false stuff.
Some of those practices should include tracking various metaphorical forests-vs-trees, and being some-kind-of-intentional about what things are worth arguing in what sort of ways. (This does not come with any particular opinion about what sort of ways are worth arguing what sort of things, just, that there exist at least some patterns of nerdy pedantry that do not automatically get to be treated as actively good parts of a good truthseeking culture)
(I think this was fairly obvious and that you are indeed being kind of obnoxious so I have strong downvoted you in this instance)
Thank you for clarifying.
No, it was not obvious!
You replied to a comment that said, verbatim, “what we should indeed sacrifice is our commitment to being anal-retentive about practices that we think associate with getting the precise truth, over and beyond saying true stuff and contradicting false stuff”, with, “This paragraph feels righter-to-me”.
That response does prompt the reader to wonder whether you believe the quoted statement by Malcolm McLeod, which was a prominent thesis sentence of the comment that you were endorsing as feeling righter-to-you! I understand that “This feels righter-to-me” does not mean the same thing as “This is right.” That’s why I asked you to clarify!
In your clarification, you have now disavowed the quoted statement with your own statement that “We absolutely should have more practices that drive at the precise truth than saying true stuff and contradicting false stuff.”
I emphatically agree with your statement for the reasons I explained at length in such posts as “Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think” and “Heads I Win, Tails?—Never Heard of Her; Or, Selective Reporting and the Tragedy of the Green Rationalists”, but I don’t think the matter is “fairly obvious.” If it were, I wouldn’t have had to write thousands of words about it.
Yeah, I kind of regret not prefacing my pseudo-review with something like this. I was generally writing it from the mindset of “obviously the book is entirely correct and I’m only reviewing the presentation”, and my assumption was that trying to “sell it” to LW users was preaching to the choir (I would’ve strongly endorsed it if I had a big mainstream audience, or even if I were making a top-level LW post). But that does feel like part of the our-kind-can’t-cooperate pattern now.
This is an absolutely fantastic phrasing/framing.
I’ll say (as a guy who just wrote a very pro-book post) that this vibe feels off to me. (I’m not sure if any particular sentence seems definitely wrong, but, it feels like it’s coming from a generator that I think is wrong)
I think Eliezer/Nate were deliberately not attempting to make the book some kind of broad thing the whole community could rally behind. They might have done so, but, they didn’t. So, complaining about “why our kind can’t cooperate” doesn’t actually feel right to me in this instance.
(I think there’s some kind of subtle “why we can’t cooperate” thing that is still relevant, but, it’s less like “YOU SHOULD ALL BE COOPERATING” and more like “some people should notice that something is weird about the way they’re sort of… ostentatiously not cooperating?”. Where I’m not so much frustrated at them “not cooperating,” more frustrated at the weirdness of the dynamics around the ostentatiousness. (This sentence still isn’t quite right, but, I’mma leave it there for no)
IMO they missed their opportunity and now LW is missing its/our opportunity, and either side naturally thinks it’s more the other’s fault.
Keep in mind propagandizing it is also an easy way to get political polarization.
Indeed. This is why one might choose a different book title than “If Anyone Builds It, Everyone Dies”.
EDIT: On reflection, I retract my (implicit) claim that this is a symmetric situation; there is a difference between what you say unprompted, vs what you say when commenting on what someone else has said. It is of course still true that one might choose a different book title if the goal was to build around areas of agreement.
My impression of the lesson from the Shanghai Communique is not “parties should only ever say things everyone else will agree with them on” but rather “when talking to broad audiences, say what you believe; when attempting to collaborate with potential partners, build as much collaboration as you can on areas of agreement.”
I don’t have much interest in trying to speak for everyone, as opposed to just for myself. Weakening the title seems to me like it only makes sense in a world where I’m trying to represent some sort of intersectional view that most everyone agrees upon, instead of just calling it like I see it. I think the world would be better off if we all just presented our own direct views. I don’t think this is in tension with the idea that one should attempt to build as much collaboration as possible in areas of agreement.
For instance: if you present your views to an audience and I have an opportunity to comment, I would encourage you to present your own direct views (rather than something altered in attempts to make it palatable to me). Completely separately, if I were to comment on it, I think it’d be cool of me to emphasize the most important and relevant bits first (which, for most audiences, will be bits of agreement) before moving on to higher-orsee disagreements. (If you see me failing to do this, I’d appreciate being called out.)
(All that said, I acknowledge that the book would’ve looked very different—and that the writing process would have been very different—if we were trying to build a Coallition of the Concerned and speak for all EAs and LessWrongers, rather than trying to just blurt out the situation as we saw it ourselves. I think “I was not part of the drafting process and I disagree with a bunch of the specifics” is a fine reason to avoid socially rallying behind the book. My understanding of the OP is that it’s trying to push for something less like “falsely tell the world that the book represents you, because it’s close enough” (which I think would be bad), and more like “when you’re interacting with a counterparty that has a lot of relevant key areas of agreement (opening China would make it richer / the AI race is reckless), it’s productive to build as much as you can on areas of agreement”. And fwiw, for my part, I’m very happy to form coalitions with all those who think the race is insanely reckless and would be better off stopped, even if we don’t see eye to eye on the likelihood of alignment success.)
On reflection I think you’re right that this post isn’t doing the thing I thought it was doing, and have edited my comment.
(For reference: I don’t actually have strong takes on whether you should have chosen a different title given your beliefs. I agree that your strategy seems like a reasonable one given those beliefs, while also thinking that building a Coalition of the Concerned would have been a reasonable strategy given those beliefs. I mostly dislike the social pressure currently being applied in the direction of “those who disagree should stick to their agreements” (example) without even an acknowledgement of the asymmetricity of the request, let alone a justification for it. But I agree this post isn’t quite doing that.)
(Fwiw, I personally disclaim any social pressure that people should avoid mentioning or discussing their disagreements; that’d be silly. I am in favor of building upon areas of agreement, and I am in favor of being careful to avoid misleading the public, and I am in favor of people who disagree managing to build coalitions, but I’m not in favor of people feeling like it’s time to stfu. I think the “misleading the public” thing is a little delicate, because I think it’s easy for onlookers to think experts are saying “i disagree [that the current situation is reckless and crazy and a sane world would put a stop to it]” when in fact experts are trying to say “i disagree [about whether certain technical plans have a middling probability of success, though of course i agree that the current situation is reckless and crazy]”, and it can be a bit tricky to grumble about this effect in a fashion that doesn’t come across as telling people to stfu about their disagreements. My attempt to thread that needle is to remind people that this misunderstanding is common and important, and thus to suggest that when people have a broad audience, they work to combat this misread :-))
I also disclaim this social pressure! Seems pretty bad IMO (and I have commented myself on the linked tweet thread saying so)
ty! yeah. tbc i would also not endorse “falsely tell the world that the book represents you, because it’s close enough” but i do think when parties have ~some reason to want to be cooperative, it is productive to build on areas of agreement, and i felt that has been missin
I’m glad you wrote this, I very much feel the same way but I wasn’t sure how to put it. It feels like many reviewers—the ones who agree that AI x-risk is a big deal, but spent 90% of the review criticizing the book—are treating this like an abstract philosophical debate. ASI risk is a real thing that has a serious chance of causing the extinction of humanity.
Like, I don’t want to say you’re not allowed to disagree. So I’m not sure how to express my thoughts. But I think it’s crazy to believe AI x-risk is a massive problem, and then spend most of your words talking about how the problem is being overstated by this particular group of people.
I feel like every time I write a comment, I have to add a caveat about how I’m not as doomy as MIRI and I somewhat disagree with their predictions. But like, I don’t actually think that matters. If you think there’s a 5% or 20% chance of extinction from ASI, you should be sounding the alarm just as loudly as MIRI is! Or maybe 75% as loudly or something. But not 20% as loudly—how much you should care about raising concern for ASI is not a linear function of your P(doom).
To be fair, if you are reading reviews of IABIED on LessWrong, you are probably already pretty convinced of AI risk being a pretty big deal. But probably good to keep in mind the general vibe that we’re all on the same team
I think it’s important to remember as well that not all of us actually are on the same team. Not everyone is even on team humanity.
For instance, Nick Bostrom, I was disturbed to learn, does not want to slow AGI development, and is perfectly willing to gamble away other people’s lives without their consent. That is difficult to top, as it concerns things that could place us on different teams.
Insofar as we the concerned actually are on the same team (humanity should survive and we shouldn’t endanger its existence beyond what most people could bear if they were informed), then we should indeed act like we are here primarily to save the world, and not primarily to move words around and score internet points.
guys I cut this but honestly do u consider riding a motorcycle to be within your risk budget? would you be excited or discouraging if a friend or loved one started riding a motorcycle? do you consider building superintellgience to be within your risk budget?
I beg everyone I love not to ride a motorcycle.
Well, I also have have a few friends who clearly want to go out like a G before they turn 40, friends whose worldviews don’t include having kids and growing old—friends who are, basically, adventurers—and they won’t be dissuaded. They also free solo daylong 5.11s, so there’s only so much I can do. Needless to say, they don’t post on lesswrong.
I don’t recall the precise numbers, but I think the last time I looked them up the micromorts for motorcycles are crazy and I would definitely go out of my way to talk a friend down from buying one
Yes I made a basically similar point.
https://www.lesswrong.com/posts/RnKmRusmFpw7MhPYw/cole-wyeth-s-shortform?commentId=itJv4dntCpGQ7kwTr
It’s just not the case that there are no fundamental disagreements , only detailed nitpicks.
A lukewarmer who believes in , say, a 30% chance of dystopia just isn’t on the same.page as an extremist who believes in 98% certain doom. They are not going to support nuking data centers.
There is a common pattern of trying to bracket moderates with extremists. The anti immigrationist who wants 10% less immigration for economic reasons is likely to.find themselves bracketed with the anti immigrationist who wants 100% less for racial reasons, and so on. The thing is, it’s actually an anti pattern: it’s a bad thing we need less of.
(I honestly don’t know whether the argument here is “if you are a doomer, you should emphasise broad agreement over minor differences, if you are not, that’s fine” or “if you are a sceptic you should be, it’s irrational to be anything but a doomer”).
Eliezer doesn’t support nuking data centers, either. He supports an international treaty that, like all serious international treaties, is backed by a credible threat of violence.
(I suppose someone with a 98% P(doom) might hypothetically support unconditionally nuking data centers, but that is not Eliezer’s actual position. I assume it’s not the position of anyone at MIRI but I can only speak for Eliezer because he’s written a lot about this publicly.)
Hmm, well, that creates a paradox, because saying it ’s only a feint makes it less credible.
It’s not only a feint. You don’t want to go to war, and you hope that the treaty will prevent war from happening, but you are prepared to go to war if the treaty is violated. This is the standard way treaties work.
There are many treaties and many times treaties are violated for various reasons. Waging a war because a treaty gets violated is not the standard way.
War is not the only potential response. I don’t know why this is being framed as normal when a normal treaty would have something like sanctions as a response.
I understand that when a person feels a lot is on the line it is often hard for that person to not come across as sanctimonious. Maybe it’s unfair of me, but that is how this comes across to me. Eg “people who allegedly care”.
Death with Dignity:
This is a good insight about a possible reasoning mistake. Likewise, if more optimistic assumptions about AI are correct, you should not “step sideways” into an imaginary world where MIRI is right about everything “just to be safe”. Whatever problems come with AI need to be solved in the actual world, and in order to do that it is very very important to form good object-level beliefs about the problems
Since nobody seems to have posted it yet:
Riding a motorcycle for 60 years:
(1-1/800)^60=0.928
Sailing across the ocean every month for 60 years:
(1-(1/10000))^(60*12)=0.931
The sailing risk is probably overestimated. I have never met anyone who was lost at sea, never seen pictures of someone lost at sea, never heard back from the people who I thought might be lost at sea, and I’m sure to find shore soon and I think I have enough peanut butter for another week...
Is this feeling reasonable?
A selfish person will take the gamble of 5% risk of death for a 95% chance of immortal utopia.
A person who tries to avoid moral shortcomings such as selfishness will reject the “doom” framing because it’s just a primitive intelligence (humanity) being replaced with a much cleverer and more interesting one (ASI).
It seems that you have to really thread the needle to get from “5% p(doom)” to “we must pause, now!”. You have to reason such that you are not self-interested but are also a great chauvinist for the human species.
This is of course a natural way for a subagent of a instrumentally convergent intelligence, such as humanity, to behave. But unless we’re taking the hypocritical position where tiling the universe with primitive desires is OK as long as they’re our primitive desires it seems that so-called doom is preferable to merely human flourishing.
So it seems that 5% is really too low a risk from a moral perspective, and an acceptable risk from a selfish perspective.
I think it’s deeply immoral to take a 5% of killing everyone on earth in the next decade or two w/o their consent, even if that comes with a 95% chance of utopia.
I think that this sort of reasoning is sadly all too common.
I think there’s a certain pattern of idealistic reasoning that, I think, may have produced the most evil pound-for-pound throughout history. People say that for the sake of the Glorious Future, we can accept, must accept, huge amounts of suffering. Indeed, not just our suffering, but that of others, too. Yes, it may be an unpleasant business, but for the Glorious Future, surely it is a small price to pay?
That great novel starring the Soviet’s planned economy, Red Plenty, has a beautiful passage example of such a person.
This person has fallen into an affective death spiral, and is lost. Like the Khmer Rouge, like the witch hunters, like many other idealists throughout history, they found it oh so easy to commit the greatest of atrocities with pride.
Perhaps it is all worth it. I’m doubtful, but it could be true. However, I would advise you to beware the skulls along the path when you commend actions with a >1% chance of killing everyone on earth.
This seems too pattern matchy to be valid reasoning? Let’s try an exercise where I rewrite the passage:
Aha, I have compared AI regulationists to the Communists, so they lose! Keep in mind that it is not the “accelerationist” position that requires centralized control and the stopping of business-as-usual, it is the “globally stop AI” one.
(But of course the details matter. Sometimes forcing others to pay costs works out net positively for both them and for you...)
If you are actually confident that AI won’t will kill us all (say, at P > 99%) then this critique doesn’t apply to you. It applies to the folks who aren’t that confident but say to go ahead anyway.
I was assuming conditional on 1 in 20 chance of AI kills everyone
Basically I don’t think the anti “coercing others for ideological reasons” argument applies to the sort of person who thinks “well, I don’t think a 1 in 20 chance of AI killing everyone is so bad that I’m going to support a political movement trying to ban AI research; for abstract reasons I think AI is still net positive under that assumption”
The action / inaction distinction matters here
But they are doing things that they believe introduce new, huge negative externalities on others without their consent. This rhymes with a historically very harmful pattern of cognition, where folks justify terrible things to themselves.
Secondly, who said anything about Pausing AI? That’s a separate matter. I’m pointing at a pattern of cognition, not advocating for a policy change.
The comment you were criticizing stated
This comment seems more to be resisting political action (pause AI) than pursuing it. If anything, your concern about political actors becoming monsters would more apply to the sort of people who want to create a world government to ban X globally, than people bringing up objections.
https://ifanyonebuildsit.com/5/why-dont-you-care-about-the-values-of-any-entities-other-than-humans
Soares is failing to grapple with the actual objection here.
The objection isn’t the universe would be better with a diversity of alien species which would be so cool, interesting, and {insert additional human value judgements here}, just as long as they also keep other aliens and humans around.
The objection is specifically that human values are base and irrelevant relative to those of a vastly greater mind, and that our extinction at the hands of such a mind is not of any moral significance.
The unaligned ASI we create, whose multitudinous parameters allow it to see the universe with such clarity and depth and breadth and scalpel-sharp precision that whatever desires it has are bound to be vastly beyond anything a human could arrive at, does not need to value humans or other aliens. The point is that we are not in a place to judge its values.
The “cosmopolitan” framing is just a clever way of sneaking in human chauvinism without seeming hypocritical: by including a range of other aliens he can say “see, I’m not a hypocrite!”. But it’s not a cogent objection to the pro-ASI position. He must either provide an argument that humans actually are worthy, or admit to some form of chauvinism, and therefore begin to grapple with the fact that he walks a narrow path, and as such rid himself of the condescending tone and sense of moral superiority if he wishes to grow his coalition, as these attributes only serve to repel anyone with enough clarity-of-mind to understand the issues at hand.
And his view that humans would use aligned ASI to tile the universe with infinitely diverse aliens seems naive. Surely we won’t “just keep turning galaxy after galaxy after galaxy into flourishing happy civilizations full of strange futuristic people having strange futuristic fun times”. We’ll upload ourselves into immortal personal utopias, and turn our cosmic endowment into compute to maximise our lifespans and luxuriously bespoke worldsims. Are we really so selfless, at a species level, to forgoe utopia for some incomprehensible alien species? No; I think the creation of an unaligned ASI is our only hope.
Now, let’s read the parable:
The odds of a mind infinitely more complicated than our own having a terminal desire we can comprehend seem extremely low.
Oh, great, the other character in the story raises my objection!
Let’s see how Soares handles it.
Oh.
He ignores it and tells a motte-and-bailey flavoured story about an AI with simple and low-dimensional values.
Another article is linked to about how AI might not be conscious. I’ll read that too, and might respond to it.