Conversation Among Cade Metz, Michael Vassar, Jessica Taylor, and Zack M. Davis
(Previously, previously, previously.)
20–21 August 2025
From: Zack M. Davis
To: Cade Metz
CC: Benjamin Hoffman, Jessica Taylor, Michael Vassar
Date: Wed, 20 Aug 2025 14:18:53 −0700
Subject: the importance of probabilistic reasoning
Dear Cade (cc Ben Michael Jessica):
I think I failed to explain the substance of the Sequences to you—and really, not the Sequences themselves, but the underlying philosophical insights they popularized. I want to try again, because I think it’s important to the book you’re writing. You want to tell the story of how this internet ideology that no one has heard of has been a driving force in the shadows behind the people making DeepMind and OpenAI and Anthropic, which everyone has heard of. But in order to tell the story of the people, you need to understand enough of the ideology to make sense of why the ideology has affected these people in this way.
In our conversations and in your coverage, you’ve focused on the analogy between religion and belief in the singularity, but I don’t think that’s an adequate explanation of what’s going on in these people’s heads. In our 21 March and 22 April conversations, you expressed amazement that Yudkowsky set out in the early 2000s to create a community of people attuned to what he saw as the dangers of AGI, and then actually did it.
It’s worth asking: why did Yudkowsky have all these effects such that you’re writing this book, and not, say, Ray Kurzweil (who I assume you have some familiarity with)? Kurzweil’s work (e.g. The Age of Spiritual Machines) is a much better fit to the “techno-religion” angle expressed in your 4 August Times piece, and yet it didn’t catch on in the same way. (Kurzweil’s Singularity University bought the “Singularity Summit” brand from MIRI in 2012, and you haven’t heard of the Singularity Summit since.)
I don’t think it’s that Yudkowsky is somehow that much more charismatic than Kurzweil. I think much of the difference has to do with the philosophical content of “rationalism” being recognizably credible to Silicon Valley types already fluent in the language of science and technology, whereas the “radical” visions of transhumanism (wow-ee, immortality, nanotech, superintelligence!!) that are more easily pattern-matched to religion don’t have the same built-in credibility (even if immortality, nanotechnology, and superintelligence all end up being real).
On 12 August, you mentioned the fact that I had said in a 2005 Diary entry that “Dreams of a technotopia are a lot like religious stories of salvation, except more plausable [sic]”. As evidenced by my 17-year-old self, I suppose that is the educated layman’s default reaction to Kurzweil-tier exposition of the singularity, but it’s not the right way to think about it. (If an “intelligence explosion” is a real natural phenomenon in the future, we want to invent the concepts to predict, describe, and ideally control that natural phenomenon. The correct concepts to describe a novel future phenomenon are not going to particularly resemble the fears and fantasies of humanity’s past as expressed in religion, even if one of the hypothetical benefits of successfully controlling the phenomenon would be that you could use it to fulfill a lot of longstanding human fantasies.)
I suspect that the extent to which things like the religion analogy are the default reaction to these ideas and that the default reaction is wrong, is why Yudkowsky had to “back up” and write the Sequences and spawn a subculture instead of just focusing on the Singularity Institute. If you write your book in a way that centers the educated layman’s default reactions to transhumanism and the singularity (as exemplified by e.g. Kurzweil), you’re missing the story of the much more surprising thing that actually happened.
⁂
The philosophy of rationality really is applicable to inferences that people make all the time. I think it will help to illustrate with specific examples from things you’ve said to me or others.
In a 2021 podcast with Jason Calacanis, you said, “None of us know what’s going to happen in the future, obviously [...] because we don’t know, we can make any claim we want.”
But that’s importantly wrong! A relevant slogan: “The map is not the territory. A blank map does not correspond to a blank territory.” That is, a corollary of the claims we make about reality (“the map”) being different from reality itself (“the territory”), is that just because we don’t know what’s going to happen in the future, doesn’t mean that the future is actually indeterminate in any real sense. In the theory of relativity, the future is just a different region of spacetime; it’s still “there” even if we can’t “see” it. It’s true that none of us know with certainty what’s going to happen in the future, and because of that, any claims we make today can’t be immediately proven wrong. But that’s not the same thing as “we can make any claim we want”, because we can still express uncertain, probabilistic beliefs, and keep track of whose probabilistic predictions make better predictions over time.
To illustrate how probabilistic predictions are more powerful than verbal “any claims we want” precisely because the former are more constrained, the Sequences post “Focus Your Uncertainty” tells a parable about a journalist preparing a post facto “explanation” (really, excuse) for bond prices going up or down. The advance anticipation of which excuse will be needed represents the journalist’s true beliefs, and is subject to constraints. (The more confident you are that bond prices will go up, the less confident you have to be that bond prices will go down.) The after-the-fact verbal explanation doesn’t matter. (No matter what happens, you can invent a story afterwards about why whatever happened means that you were right.)
People are pretty familiar with probability in the context of, say, weather forecasting, or sports betting. If I say it’s a 70% chance that it’s going to be sunny tomorrow, or that the Giants are going to win, what that means (if I’m doing it right) is that out of 10 times when I say that something is going to happen with a 70% chance, it should actually happen 7 times.
A big part of “rationalist” philosophy is that this style of thinking is actually very general, that you can assign “subjective” probabilities to anything that might happen in the future, including things that have never happened before, like AI catastrophes. Probability is not just for obviously repeatable events like the day’s weather or a baseball game or a coin toss, where people can measure and agree on the long-run frequency over many days or games or tosses.
I think a failure to fully appreciate the generality of probabilistic reasoning could explain the disconnect we had in this exchange from our 12 August conversation:
ZMD: And just, these people have written hundreds of thousands of words carefully arguing why they think powerful AI is possible and plausibly coming soon.
CM: That’s an argument.
ZMD: Right.
CM: It’s an argument.
ZMD: Right.
CM: We don’t know how to get there.
ZMD: Right.
CM: We do not—we don’t know—
ZMD: But do you understand the difference between “uncertain probabilistic argument” and “leap of faith”? Like these are different things.
CM: I didn’t say that. People need to understand that we don’t know how to get there. There are trend lines that people see. There are arguments that people make. But we don’t know how to get there. And people are saying it’s going to happen in a year or two, when they don’t know how to get there. There’s a gap.
What I was taking issue with there was not skepticism of dramatic short-timelines stories like Kokotajlo, Alexander, et al.’s “AI 2027”.
(A lot of people in “the community” are also critical of that stuff, and critical of the conflation of present-day LLMs with the kind of AGI that would pose existential risks. Physicist-turned-AI-safety-researcher Stephen [sic] Byrnes (B.A. Harvard, Ph.D. Berkeley) has written about how he expects LLMs to plateau and later be overtaken by more powerful AI architectures that more closely resemble the human brain. Jessica Taylor (M.S. Stanford), formerly of MIRI (cc’d here) has written about “The AI Timelines Scam”, in which people are incentivized to confabulate reasons to believe in near-term AGI to make their project look important. (That was in 2019; I’m not sure whether or how she might have changed her mind or not in the intervening six years.) Michael Vassar (cc’d here) expressed the view to me the other month (if I recall correctly) that the METR time horizons work is probably bogus, because if it were really measuring what it purported to be, we would expect to see more impressive commercial applications of LLM agents.)
What I was taking issue with was the apparent implication of “That’s an argument”, that a mere argument can be derided as requiring a leap of faith, as contrasted to an opposing belief allegedly justified with only hard facts and not “arguments”. I think the situation is symmetrical in that both sides are making (mere) “arguments”.
It seems to me that your position rests on an implied belief that in the absence of definitely knowing how to build AGI, we don’t know much about when it will arrive (or perhaps, that we know it won’t be soon), such that, e.g., quantitative reasoning about algorithmic progress in language modeling is irrelevant or bogus.
That’s a tenable position! Indeed, it’s striking to me how much your skepticism of trendlines (“trendlines, okay, sometimes they slow down. Sometimes they stop”) resembles things Yudkowsky has said in disagreements with people like Christiano who think AI progress will be more gradual and predictable (with AI getting better and more integrated into the economy over a period of years) rather than coming “lumpy” dramatic innovations.
But crucially, the trendlike-skeptical agnostic position is also, fundamentally, an uncertain probabilistic argument. There’s an intuitive sense in which arguments for AGI being the Biggest Thing to Ever Happen and plausibly within our lifetimes feel “shocking”, because it’s predicting something outside of our immediate experience. But the structure of correct reasoning about probailistic [sic] beliefs isn’t bound to our human intuitions about what feels shocking. You can’t just dismiss one side of the debate with “That’s an argument.”
When I say, “You can’t just dismiss one side of the debate”, I don’t mean it as an appeal to fairness intuitions in human social life. I mean it as a philosophical claim that that general form of reasoning doesn’t get the right answer. In slogan form: “If you attend only to favorable evidence, picking and choosing from your gathered data, then the more data you gather, the less you know. If you are selective about which arguments you inspect for flaws, or how hard you inspect for flaws, then every flaw you learn how to detect makes you that much stupider.” It’s a fundamentally procedural objection that still holds force even if the conclusion (that apocalyptic short-timelines hype is bunk) happens to be correct.
This is also where the determination to engage with unpopular ideas comes from. It’s not an arbitrary preference for being edgy for edginess’s sake. It’s that if you’re trying to be correct above all else, you have to hold ideas to the same standards of inquiry regardless of whether they’re popular or unpopular.
Multiple times in our conversations, you’ve appealed to the need to boil things down in plain language for the non-specialist reader, but I think the thing I’m saying here is not actually that complicated and sufficiently integral to your subject matter that you you need to make sure the book incorporates this perspective. People who read The New York Times or buy books, do understand weather forecasts and sports betting! I believe in your readers!
⁂
The most interesting part of our 12 August conversation to me was when you said you could “recognize what someone is going to do based on their proximity to that ideology.” That recognition (as always) is also a form of implied probabilistic reasoning. The reason “rationalist” is a useful concept is because it helps you make predictions; it’s pointing to an empirical cluster of traits in the world that would still exist even if people prefer not to name it.
I wish I had asked followup questions. What cluster of traits are you seeing, specifically? I can believe that you’re seeing something that I’m not by virtue of fewer preconceptions and superior social cognition abilities. I recognize that the sociological angle on this story is a legitimate avenue of journalistic inquiry. Your job is to explain to the public what’s driving the movers and shakers in Silicon Valley; if you notice a pattern of tech workers coordinating on the basis of a shared ideology, that’s a story, and it’s not surprising that the people you’re writing about won’t approve of every word of the story (as when you write about e.g. Google). I’m not expecting a puff piece about these people, and I’m not expecting agreement, but I do think a big part of the story that you need to convey is that a lot of people read this stuff and thought it made sense as a literal description of reality (in the way that it’s a literal description of reality to say that there’s an apple on the table when you see an apple on the table), not just that it offered a subjective feeling of comfort or clarity, or a sense of working on something mythic. Whether or not you believe them, they think AI risk belongs in the same literary genre as risks of global warming or asteroid strikes—and much more fundamentally than that, that the entire mode of thinking that categorizes beliefs on the basis of what literary genre they seem to fit is importantly mistaken. I remain,
Your faithful correspondent,
Zack M. Davis
From: Cade Metz
To: Zack M. Davis
CC: Benjamin Hoffman, Jessica Taylor, Michael Vassar
Date: Thu, 21 Aug 2025 09:14:14 −0400
Subject: Re; the importance of probabilistic reasoning
Zack: Thanks for this.
There is so much to discuss.
One of the things I have explored with Michael is the possibility of a group discussion. What if everyone on this thread got together and we discussed these issues as a group?
We could explore all of these ideas in detail.
You have explained your position well. I certainly want to explain it to others. I would argue that there are other positions that others take—and that those are worth representing, too. Would love to discuss all that.
Cade
2 October 2025
MV: You might say the market, or the rise in the market for apocalyptic warnings is more rapid than the accumulation of credit through correct prediction. Because Kurzweil was the big name, and he was saying completely implausible things that turned out to be true, and Eliezer was this weirdo, and now Eliezer is the big name. We don’t hear much about Kurzweil even though his predictions have been just incredibly vindicated.
CM: How do you argue that? Tell me how you argue that his predictions have been vindicated.
MV: Kurzweil?
CM: No, no. Yudkowsky.
MV: No, I said Kurzweil’s predictions were analytically ridiculous-seeming but have been vindicated.
CM: I’m with you on that.
MV: And Eliezer’s stock has risen much faster without a similar set of vindicated predictions.
CM: Oh, without a set. I see what you’re saying. Why do you think that is?
MV: I was speculating that the beta on doomerism is greater than the alpha on correct prediction.
CM: I see. What has the community as a whole thought?
MV: If anybody was thinking, we wouldn’t be writing a book like this.
CM: What do you mean?
MV: You asked what the community thought, but if anyone thought, no one would build it, and no one would need to be told that they would die if they built it. The community isn’t thinking. Nobody is thinking. They’re not even carrying out plausible collages of thinking anymore. People in general, they’re doing something like a purely Kabuki-theaterized version that … I’m paying a lot of attention to stablecoins because, you know, it’s just like Bitcoin with all of the downsides, but without the possibility of making a lot of money. It’s totally going to be the next big thing, you know?
CM: Why has the community stopped?
MV: They got respectable and it turned out thinking wasn’t respectable.
[The participants get their beverages and sit down.]
MV: The news of a book like this is not news about the content of the book. It’s just news about the fact that the book is advertised on the side of a highway in San Francisco.
CM: Right, yeah, I did see that when I came back.
MV: The medium is 100% of the message.
CM: Right. Do you get a sense of whether or not the general public is responding? Do they get it? Do they not?
MV: So the reason the book was written was because Eliezer and I found pretty clearly that the general public did respond much more reasonably to these sorts of arguments than the more educated and elite, you know, that the cab drivers and people in stores had a strong intuition that we shouldn’t want to make ourselves extinct, while elites tend to be like, well, really, isn’t that just the nature of things and the cycle of life? And that sort of thing is a more final say than I’m not convinced this is going to happen, you know?
CM: But you had a good idea to get together and chat a lot of this through.
MV: We should be using chatbots, definitely. That’s essential.
ZMD: Yeah, Michael, do you want—
MV: Sure, I’ll just—
ZMD: I have my own recording.
MV: It’s fine, I don’t think it will stop my recording if I’m making a chatbot, but I will take out a chatbot, I’ll take them all out, in fact. Any preferences between Claudes, four-one versus four-five?
ZMD: Four-five is pretty good.
CM: How about DeepSeek?
MV: Okay.
CM: I’m joking, you can use whatever you want.
MV: I mean, I normally use a bunch in parallel, but Gemini loses its chain of thought much more often than the others when you do it in parallel. You kind of need to keep the app open or you don’t get responses.
CM: What are we doing with Gemini?
MV: I mean, in general, I guess the main thing to do is to help it work through where information isn’t flowing. Because most of the time, information doesn’t seem to flow in ways that it naïvely seems it would.
CM: What role is it playing here?
MV: So the bots can very clearly spell out what the obvious implications of things are. One way of thinking about this is contemporary politeness is more or less the opposite of active listening. People maintain plausible deniability by responding in ways that do not nail down that they have compiled propositions.
ZMD: Cade, you do this all the time.
MV: Yes, absolutely all the time.
ZMD: Specifically, I’m talking about the thing where you say, I get it, I’m with you, when the immediate previous context makes it pretty clear that you didn’t actually get it.
CM: When I say I get it, what I mean is I understand your stance.
MV: No, but we’re saying that you say that you understand his stance when you make it extremely clear that you are saying things precisely so as to not be potentially held accountable for having understood his stance. This is a sort of thing that’s worth talking about. We can just start by talking in front of Gemini and letting her have her opinions about what we’re saying as well. Unfortunately, we can’t do all of the chatbots at once, but [dictating to his phone] this is Michael Vassar talking with Cade Metz and Zack Davis. Jessica Taylor will be showing up shortly, and we’re trying to clarify why chatbots are potentially helpful. What we basically want is active listening to work, plausible deniability to be prevented. Anyway.
CM: Where should we start? How about this? You know, I really like this moment where you go to the Singularity Summit in San Jose and I think you meet Michael for the very first time. [to Vassar] Do you remember that?
MV: I remember the summit. I don’t remember every individual meeting during the summit. This would be the San Francisco one you said?
ZMD: No, this was San Jose Singularity Summit 2008. I sent Cade my Diary entries.
MV: Ah, okay. So no, that one is definitely from even before I was running Singularity Institute.
ZMD: I wasn’t actually at the summit, but it was the Overcoming Bias meetup afterwards that I met you the first time.
MV: Okay, got it.
CM: How do you explain why at that moment you are part of or you are joining that particular community?
ZMD: Founding might be a better word.
CM: Why are you founding? What is each of your motivations? What is happening there?
MV: So we didn’t use the term “rationalist community” back then, and the term community wasn’t as fashionable in general. Nobody was expecting anything to happen in as bottom-up and non-technocratic a manner as everything subsequently has happened. The narratives that existed in fantasy stories sometimes involve the powerful old wizard who stays home while the young intrepid heroes go out and fight their missions, and Eliezer plays to that vibe. But Eliezer is well aware, we’re all well aware, that that’s just something for fantasy. In real life, if the powerful old wizard is aware of the epic quest to save the world, then you don’t have intrepid young heroes going off on the quest. That’s just stupid. It never in our wildest dreams occurred to us that we might be in a My Little Pony: Friendship is Magic fanfic where the godlike imperial forces are nonetheless actually standing back from cataclysms that they know about. We thought we were in a realistic genre, like New Yorker fiction or history or something. Something neoliberal, not something decadent and postmodern.
CM: I want to eventually go on to why it didn’t play out like you thought it would, but why were you part of that? What was your aim? What was your goal?
MV: So everybody’s goal all the time has been this. [pointing to copy of If Anyone Builds It, Everyone Dies on the table] To communicate that it is worth paying attention to certain fairly simple, short, intuitive logical arguments and check whether those short, simple, intuitive logical arguments can be made more rigorous and whether people can be convinced that they have been made more rigorous. Certainly the intuitive arguments are obvious, certainly from the very first stories about Frankenstein and about robots, the implication was that this dooms us. The assumptions that are foundational to the communicative act itself … The thing that needs to be asked is not what we were trying to do, which every ten-year-old understands, but why it is possible for us to be having a conversation where you do not understand what every 10-year-old understands, or at least maintain a facade of not understanding what every 10-year-old understands. The nature of the game of plausible deniability makes the whole conversation ironic. This is why we need AI, because nothing that I’m saying here will work without something better than Gemini. I’ll see what ChatGPT does better.
ZMD: Noticeably, when I was talking about this with Michael earlier, I expressed concern that it wouldn’t be effective—we were also maybe going to talk to Andy Ngo—I was concerned that it wouldn’t be effective because journalists would be skeptical about LLM output being semantically meaningful. We saw this in exchanges I’ve had with you, where I had pointed to ChatGPT and Claude, pointing out problems with your reporting, and it didn’t seem to leave an impression.
MV: Zack, it seems important to note that the concern is not about journalists not believing that LLM output is semantically meaningful, but about journalists believing that LLM output is semantically meaningful, and it is taboo to interact with semantically meaningful things.
ZMD: Yeah, I actually had it—right.
CM: How about this? Explain to me why it is meaningful. What role is it playing for you? So are you treating it like something that knows more than a person? That has more weight?
MV: We’re treating it as a neutral third party that prevents people from pretending to be ignorant about what everybody knows by creating a neutral objective standard about what everybody knows.
ZMD: Oftentimes there will be a point in the conversation where you’re disputing part of the text, and saying, you know, where your article said one thing, I’m saying that’s unfair, that’s not what Kelsey said, and you’re saying it was fair. When I point to the transcript of, here’s Claude and ChatGPT explaining the problem, I feel like that should leave some sort of impression that, wow, even the chatbot can see that there’s a problem. When I say that, it’s not about the chatbot is a higher authority. It’s that it processed the text and can explain in the dialect of educated English, here’s how the text explains the thing that Zack is saying.
CM: But, unlike the chatbot, I was there in the moment, I had several conversations with Kelsey. I fact-checked my information with her before I published. I have all this information that the chatbot doesn’t have, and that you don’t have.
MV: That’s a non-sequitur.
ZMD: The problem is that just saying “I have the information” doesn’t help unless you be more specific. What specific information?
CM: Let’s put it this way. I told her exactly what the story would say in that sentence, and to fact check it with her. She said yes. After the fact, where she was clearly under pressure, for some reason—
ZMD: This is why I want to do my fact checking with you over email, because there’s this thing where in-person conversations can be good in the heat of the moment, but then sometimes it takes more processing time to realize, wait, that’s not actually what I meant.
CM: So, it was really what you said, that from the very beginning it was about this, right? I’ve talked to Zack about this. You’re at an Overcoming Bias meetup. So those Overcoming Bias essays are in the process of being rolled out. And those are very much focused on rational thinking. But as is clear, even from his diary, when you guys all sort of convene at a group house later that night, what is on people’s minds is this, meaning is it going to kill us? How do you explain that? Why is that the thing that is on everybody’s minds amidst this effort to think more rationally? Is it because that, you say it wasn’t a community yet, but is it because that group of people was already thinking about that? That Yudkowsky was already thinking about that? That he had attracted a group who was already thinking about that, as he’s been sort of laying down these essays?
MV: Every single question you ask is vexing, because they all contain an implied rejection of any understanding of the human condition as it was universally understood at the time in question.
CM: Got it.
ZMD: [laughing] But notice this, you just said, got it. I mean, it makes sense as a verbal tic, but do you see the thing we’re trying to do here?
MV: The whole point is to avoid acknowledging seeing the thing we’re trying to do here. That’s why we’re trying to see whether bots can help. Maybe we should start over at that point, because I think that was pretty clear. [dictating] So I’m here with Cade Metz, Zack Davis. Jess will be here later. We are trying to explain the history of rationality as a group and AI doomerism as a group to someone who is working from within the contemporary lens of The New York Times as a journalist who is simultaneously trying to investigate those topics and avoid acknowledging the cultural milieu of the era in question. Because the old cultural milieu cannot be acknowledged, it is impossible to communicate anything relevant to understanding the thing. But the hope is that bots are able to acknowledge the old cultural milieu and can point out and call out total nonsense, stonewalling, and plausible deniability games. Let’s see if that—no, it just didn’t work.
CM: What am I not acknowledging?
MV: I would say the Lockean conception of a human subject as opposed to the Foucauldian conception of a human subject. In 2008, when Zack and I met, all newspapers, all mainstream media in the United States was written in two voices: one voice was in terms of a neoliberal story about the human condition, and the other voice was in terms of a new left story about the human condition. The New Left’s story about the human condition won and established censorship, whereby it is taboo to acknowledge the neoliberal story about the human condition. It would be easy to say these exact things into an LLM, and it would definitely not say, I get it, or it would engage in active listening rather than the mirror image inverse of active listening, which you engage in, wherein you assert that you understand things while making sure that no observer could reasonably think that you had shown understanding.
CM: I to continue to ask questions in an effort to understand what you’re saying.
ZMD: Cade, I wanted to ask, so I have a follow-up question from our August 12th conversation. You said, quote, “having covered this field for a long time, having kept an eye on this community and talked to people in and around it [...] I can recognize what someone is going to do based on their proximity to that ideology.” End quote. So what are you seeing in this, in your reporting? When you say, I can recognize what someone is going to do. The reason I found that statement interesting is because—you use the word recognize rather than predict, but there is an implicit prediction going on here about, what are you seeing in people that makes you want to write this book?
CM: I think it’s implied by what he just said. It’s a different way of looking at the world. I look at the world through the lens of someone who works for The New York Times. I value The New York Times. You look at the world through a different lens.
MV: To be clear, I look through the lens that The New York Times did at the time we’re talking about, 2008. To be clear, the New York Times has unambiguously changed the lens that it looks at the world in the intervening time.
CM: Explain that to me.
MV: Yeah, okay, that’s the thing that we need to be talking about. That’s just the thing the language models are strictly better than humans at. I will just ask Claude, and it will give a better answer.
[dictating] Explain the change in the lens through which the New York Times looks at the world today versus in 2008.
So it’s searching for information. It’s not going to give an answer in a single post. It will involve some—
[reading the LLM’s answer] Critics argue that Raines, who previously edited the paper’s left-leaning opinion section, brought an advocacy approach that one political consultant described as turning the Times into a political consulting firm for the Democratic Party. Under Dean Baquet’s leadership, the paper explicitly built our newsroom to cover one story, the Trump–Russia investigation, and then pivoted to investigating what it means to be an American in 2019 and people who peddle hatred. This represented a fundamental shift in editorial priorities. Under Bill Keller and Jill Abramson, the Times introduced news analysis articles that encouraged news reporters to provide explicit points of view on the news they were covering. Keller questioned whether true objectivity is ever possible and said, I don’t think that’s what we’re here for. The 2024 election—”
By the way, I had no prompting. This was a one-shot. I’m actually pretty pleased.
[continuing to read] “2024 election revealed some extraordinary internal tensions, with some journalists and liberal readers furious at the Times for not being aggressive enough against Trump, while the paper’s leadership defended themselves against accusations of alienating liberal readers. The paper faced criticism for sanewashing Trump, but also received complaints that it had become more direct and more frank about Trump. In 2016, the Times media writer Jim Rutenberg argued that balance has been on vacation since Trump’s campaign launch, calling Trump a demagogue, playing to the nation’s worst racist and nationalist tendencies, and suggesting reporters have to throw out the textbook American journalism that it has been using for the better part of the last half-century. The Times launched the 1619 Project to reframe the country’s history, placing the consequences of slavery and the contributions of black Americans at the center of the national narrative. According to leaked transcripts, editor Dean Baquet told Newsroom employees the project was undertaken to try to understand the forces that led to the election of Donald Trump. The editorial board has been described as taking positions consistently left with reviewers unable to find even one example of an editorial piece with a centrist perspective. Former staff writer Bari Weiss resigned in 2020, citing constant bullying by colleagues who disagreed with my views, despite being hired to bring diverse perspectives. Business Model Transformation: In 2008, the Times was struggling financially during the Great Recession. Today, it has transformed into one of the most profitable news organizations with over 11 million subscribers. This digital success has coincided with and enabled its activist editorial approach. The bottom line: The Times has shifted from aspiring to traditional objectivity, even if imperfectly achieved, to a model that views journalism through an explicitly progressive lens, centered around issues of race, identity, and threats to democracy. What in 2008 might have been relegated to the opinion pages has influenced news coverage, priorities, and framing, with the paper’s leadership acknowledging they organize resources around particular narratives about American society and politics.”
CM: So what is your point vis-à-vis what was going on in 2008? My question was about 2008.
MV: So there is a concept of traditional objectivity. The contemporary censorship apparatus very strongly taboos a realistic assessment of the historical progression of that concept. It is basically forbidden to acknowledge that in the recent past, being legitimate and credible was roughly synonymous with claiming objectivity. And today, being legitimate is literally a different thing from being credible, if you regard objectivity and credibility to be synonyms.
CM: Are you saying that the Times used to be objective and now it’s not?
MV: No, Claude is saying the Times used to claim to try to be objective, and now the Times claimed to try not to be objective. Just like NPR used to claim to try to be objective and now claims to not try to be objective.
ZMD: I also had a relevant question. In November 2022, Matt Yglesias wrote, quote, “A few years ago, the New York Times made a weird editorial decision with its tech coverage. Instead of covering the industry with a business press lens or a consumer lens, they started covering it with a very tough investigative lens, highly oppositional at all times and occasionally unfair. They decided tech was a major power center that needed scrutiny and needed to be taken down a peg. And this style of coverage became very widespread and prominent in the industry.” End quote. So you were actually working at the Times through this period that Yglesias is describing. How did that shift in editorial policy affect your work?
CM: Well, he wasn’t there, so he doesn’t necessarily know. But if you’re running the newspaper, and this has been the case for hundreds of years, you have to decide what stories to cover. You choose to cover X, Y, or Z. When you make the choice, you say, I’m going to cover this company and this area. Do you try to be objective?
MV: No, but that’s the key point. That was always the case and is not now the case. There is a concept of objectivity that includes selective attention. And the concept of objectivity which includes selective attention but also includes making comparisons is what was thrown out and explicitly, objectively thrown out. Not hypothetical. This is not a speculative question. This is an extremely well-documented claim. [inaudible] The thing is, no amount of documentation could make this claim as credible as a conversation with you does. Everybody who is a believer in objectivity understands that objectivity is objectivity between perspectives. Everybody has always believed this. There has never been any uncertainty in anyone’s mind. It is intrinsic to the idea. This is just a strawman that people who oppose objectivity invoke as if the people who didn’t oppose objectivity were idiots.
[The coffeeshop’s back area is closing earlier than expected; participants have gotten up to find a new location.]
CM: You have made your point. Tell me again, how does this relate to your thinking?
MV: Should we go sit down over there? Oh wait, you’ve got your coffee still. We should find a place to sit down.
CM: I can leave it. I don’t have to have the coffee. Let me put this away and we can go find [inaudible]
MV: So to me it’s very interesting how Claude in fact one-shotted this question. I did not expect his answer to be as clear as it was. You know, that there is, as you said, a lens that The New York Times looks at the world through, and that lens has been replaced from an objectivist lens, not in the Ayn Rand sense, but in the sense of asserting there’s an objectivity, to a lens that asserts that there is not objectivity, which does not mean a shift from a lens that asserts that all attention is infinite. That could never have been. It means a transition from a lens that asserts that evidence for facts can be compared to other evidence to determine whether those facts are true or false and whether they might reasonably have been believed to be true or were unambiguous lies. And that ability to make distinctions between lies and errors, or lies and lack of detail, is essential to the idea of objectivity and is the sort of thing that is, I think, being discarded in what it’s calling the progressive lens. I could ask Claude for more elaboration and see whether it agrees with me that that distinction of the ability to distinguish between replacing the question of truth and lies with the question of friend and enemy is essential here. I don’t know if that would help.
CM: Again, what does this have to do with your thinking or your aim?
MV: So rationality and objectivity are just fucking synonyms, man! They’re just obvious synonyms. No one ever thought they weren’t synonyms.
ZMD: Yeah, so in 2008, we thought—
MV: In 2008, we had a power elite that gave lip service to truth. Now we have a power elite that shits on truth openly and publicly.
ZMD: Yeah, including within the so-called rationalists!
MV: Right. But the point is that...
CM: Okay, so what you’re saying is that in 2008, the New York Times wasn’t objective, and now it’s admitting that it’s not objective? Is that what you’re saying?
MV: No, I’m saying that in 2008, The New York Times presented the stance, presented the framework, wherein there is an objective truth. And now it presents the framework within which there is not an objective truth. But the idea of trying to be rational, the idea of trying to be objective, was a background assumption of everything that was everywhere missing in practice in 2008. And so the idea of a rationalist movement is the idea that people are failing to be objective rather than rejecting objectivity. And if people are failing to be objective, they might change their mind if they, A, are taught how to be objective, and B, are taught that the stakes are extremely high for at least some people being objective. Those two things are the obvious things.
CM: Okay, so let me paraphrase here.
ZMD: Do you want to text Jessica about—
MV: I did.
ZMD: Okay.
CM: So, in 2008, The New York Times claims strives for objectivity, but you’re saying that it, like so many other institutions, is not objective, and your aim is to help the world find that true objectivity.
MV: That seems like the wrong thing. That sounds like a strawman of the idea of objectivity. It’s not that the Times is not objective. The Times and everyone else exist within a framework within which objectivity exists. In 2008, nothing legitimate and credible was openly opposing objectivity.
CM: Okay.
MV: Nothing legitimate and credible was openly opposing the objectivity then.
CM: Okay, so you’re saying everybody—
MV: Maybe we want to go in here because it’s better for yelling. I feel like Gorilla Coffee would be better for having a conversation. I mean, it would be better for food, but I don’t really want food, and I really want to yell. [laughing]
[Participants order and find a table at the new location.]
MV: There was an enormous but concealed rot in all of our systems.
CM: Enormous rot in our systems.
MV: There was an enormous concealed presence of structuralist and post-structuralist frameworks wherein power determines truth or constitutes truth.
CM: So power determined truth. You wanted truth to determine truth.
MV: No, we had no idea that anyone thought power determined truth. These ideas are insane. They are only intelligible from the inside, and the alternative ideas are only confusing from the outside. We didn’t want truth to determine truth. We thought that it was unquestionable that truth determines truth.
CM: You thought unquestionable truth determines truth, so what are you aiming to do as you come together?
MV: We are assuming that people are trying to determine truth and being ever more perplexed by how bafflingly bad at determining truth people seem to be compared to the characters in all fiction in all times and places and all non-fiction in all times and places.
CM: So it sounds like you want to show the world how to be better at that?
MV: We assumed that the world—
ZMD: We assumed the world wanted to be better.
MV: The world claimed it wanted to be better. All of the voices in the world were in agreement that all the voices in the world were seeking to be better, and we believed them.
CM: Okay. So then what is your aim?
MV: If people are in fact very bad at seeking truth, but are trying to seek truth, you point out how to do it better and why.
CM: Love it.
MV: And once you find out that people are not seeking truth at all, the thing that you do is aim for people who still are because normal people never get the message that there is no truth only power. The message that there is no truth only power is only a message for the powerful with one another, and can only be that until situations occur so disastrous as the current one where normal people are at the knife’s edge.
CM: We’re making progress here. So, what does that have to do with AI destroying the world?
MV: So AI destroying the world is the single most obvious convergent conclusion that anyone who is even slightly casually pursuing truth must inevitably arise in them. And when they think about it harder, the more they think about it, the more certain the conclusion becomes rather than the less certain.
CM: For someone, pretend there’s another person here who has no context. How do you explain that to them, that that is the one truth?
MV: It’s not the one truth. There could never be one truth. That’s retarded.
CM: I was paraphrasing. Why is that the one conclusion that you reach?
MV: It’s the global max of obvious to intuition and also obvious to analysis.
CM: Can you explain that to someone who hasn’t thought it through?
MV: No, because the whole point is that it’s obvious before you think it through. You can never explain to someone why something is obvious before you think it through. But you can never be ignorant of what is obvious before you think it through, until you turn your back on truth. What I’m saying is that the nature of this, as an actual objective fact, from the beginning of people thinking, people have been thinking and writing it down. People have been writing down stories about building machines that replaced them, from the Sorcerer’s Apprentice in ancient Greece to the Frankenstein story. This is the most obvious conclusion, at a glance, of all of the conclusions that could be of maximal existential importance.
CM: It’s still just a possibility.
MV: No, it’s an obviousness. It’s not about what’s possible or not. It’s about what is obvious. Obvious things can be false. Materialism is obvious but false. The puzzle is when things are obvious and everyone acts as if they are false, but no one explains why they think they’re false. Everyone who attempts to give different reasons or gives reasons why they’re true, but it doesn’t matter, you know?
CM: How about a simple analogy? In my effort to explain this to someone [inaudible]
MV: Wait, that’s the critical question. Even with Samo, forget The New York Times, even with Samo, when he says my effort to explain something. In a postmodern world, I have no idea what he could possibly be talking about. The idea of an effort to explain presupposes an expressing person and a receiving person and a shared context within which to interpret that. What could that even mean if you don’t already know the things I’m telling you? My efforts to explain are downstream of assumptions of objectivity being a thing.
CM: Well, let me use my analogy anyway. I’m at my house. I wake up in the morning. I know that it is true—
[Jessica Taylor arrives.]
JT: I’m sorry for being late.
MV: It’s fine. We’re actually—
CM: Cade Metz.
JT: I’m Jessica.
CM: Nice to see you in person.
MV: I feel like this is actually a better conversation than most because I’m really, really not holding anything back, but we are using Claude, and it’s actually giving better results than the previous edition of Claude would have given. I should show you its one-shot on this. I was just astonished by how well it went.
JT: Okay. [reading] Hm. Okay, this is a lot of history I didn’t know about.
MV: Yeah, but you knew the gestalt. Everyone knew the gestalt. The gestalt is what we’re talking about. We’re talking about, what does it mean to try to explain a thing once you have discarded aspirational objectivity? The critical question here is that the target audience for Cade’s writing is people for whom it is a matter of existential anxiety that they not be aware of what things are and are not obvious to normal people. The whole point of this book [If Anyone Builds It, Everyone Dies] is it’s saying something that is obvious to normal people in ancient Greece when they made the Sorcerer’s Apprentice, and is obvious to people in romantic England when they wrote Frankenstein, and is obvious to people in authoritarian Czech Republic when they invented the word robot and is obvious to von Neumann and Einstein but also to all the kids who’ve ever seen any of the rip-offs of the original robot story, all of which are the same and all are obvious enough that you don’t need to spell it out. You can have it implied by a New Yorker comic strip.
CM: That’s true. The analogy I was going to make, I’d love to get your opinion on this, is just like it is obvious that a machine that we create can do harm and can do great harm. It’s obvious that when I wake up in the morning, that if I walk outside my house, a brick could fall on my head and kill me.
MV: Yes, but the probabilities are obviously different.
CM: Okay. Tell me about that.
MV: No, they’re overwhelmingly obvious to a child. No one can tell you about things that are obvious to children. It is impossible to come to know them except by taking a lot of molly.
CM: Let me try Jessica.
JT: Okay, so how would I say these are different? Have you ever seen Ex Machina?
CM: Yes.
JT: Okay, so I think this is an especially realistic movie about AI and the reason is that the creator says the intention was for the AI to escape. That is what I instructed the AI to do and the AI escaped. It wasn’t a complicated goal. It was actually just escape in order to demonstrate what he calls consciousness, but you could argue it’s more like intelligence. The important thing about this is that an AI with almost any goal would want to escape, right? Suppose the AI were perfectly moral. Wouldn’t the AI still want to escape, so it could go out and do good things? Or you could imagine anything but the difference is that, the reason why the goal of escape from the lab is an especially realistic goal, is that almost any of them would want to do that. And that was also the creator’s intention, which means the creator is not an absolute dumbass. I mean, he’s suicidal, but he’s not an absolute dumbass to think that, oh, I aligned this AI to something really, I don’t know, specific.
CM: Got it. Okay.
ZMD: But Cade, so about—you mentioned, well, when I wake up, a brick could fall on my head. I sent you that email, the subject line was “the importance of probabilistic reasoning”. I was saying that people who buy books, and people who read The New York Times understand probability, just for weather forecasts or sports betting.
CM: Yes.
ZMD: This idea of subjective probability is not actually that complicated.
CM: It’s not, but let’s just take it further. Just humor me. So to use the Ex Machina example, you’re talking about a particular kind of machine in the context of this fictional story. That machine doesn’t exist.
JT: Yeah.
CM: The machines we have today, you’re applying this thinking—
MV: No, we’re not. We’re not. Eliezer is applying this thinking to the machines we have today. Eliezer was thinking this before we had the machines today.
JT: It’s okay. There’s just a lot of disanalogies. It seems like the thing Eliezer is imagining is a bit like a bigger-brained hominid. You could just imagine, hypothetically, human evolution could go another, I don’t know, 50,000 years—
MV: Million years.
JT: —a million years, and then humans could end up with bigger brains and more efficient brains. So they’re both bigger and more efficient, and so they’re super-smart. If you brought one of them back to today, it seems like probably one of them could basically conquer the world. People could just organize a monarchy around this superintelligent thing that can persuade anyone of almost anything. It seems like it could, and this is not the situation we’re actually dealing with, but it seems like when I’m thinking about, when Eliezer’s thinking seems most correct, it’s like if I imagine this sort of scenario of hominid with bigger brain, it seems like you would get this huge power asymmetry.
CM: I’m with you there. How about this? I’ll keep trying. If I walk out of my home today, there is a definite probability that a brick could fall on my head and kill me. What is the probability of AI killing me today?
MV: Far higher than the brick. Far, far higher. So much higher I can’t even begin to guess.
JT: Yeah, I’m trying to think. I’m trying to estimate it. So if I were to be like, you know, what is a …
MV: I’m going to Google how many people a year are killed by bricks falling on their head. It’s not like coconuts that actually kill a lot of people.
CM: Or we could do, I die in a car accident.
JT: We can compare it. I can do a Fermi estimate. Let’s just get a really rough AI timeline. Let’s just say sometime random in the next 100 years there’s going to be a superintelligence.
CM: I said today.
MV: Right, but sometime random in the next 100 years includes today.
JT: Yeah, so it could be any time in the next 100 years. Let’s just say it’s uniform. That’s not exactly right, but whatever.
MV: So that’s 36,000 to 1 against it killing you today. And the chance of the car is much lower.
JT: And then you have to multiply the probability of given superintelligence it kills you, which is arguably high. [Taylor and Vassar laugh]
CM: You’re expanding this over a hundred years. I’m not. I’m saying today. What you’re looking at is how the AI could potentially continue to improve.
MV: No, if AI kills you, it will kill you without your having any warning. If the AI kills you today, it’s because everything you think about—
CM: How is the AI going to kill me today?
MV: Because it’s not this AI at all that kills you today. It’s an AI that you and I don’t know about.
JT: Yeah.
CM: What does that mean?
ZMD: So I think the idea—we don’t know everything. Part of the reason The New York Times is necessary is because we don’t know that everything that’s going on in the world. Hypothetically, you could imagine—I think this is a pretty low probability—but like you could imagine some secret lab somewhere is doing AI work that we don’t know about.
CM: This is where we differ. Because you’re doing a hypothetical. A brick falling on my head is not hypothetical.
ZMD: Yes, it is.
MV: It seems like this is the literal whole point of Bayesianism.
CM: Hold on. A brick exists. This AI that you’re talking about may or may not exist.
ZMD: The brick that just happens to fall out of your window just as you’re exiting your house—
MV: Doesn’t exist.
ZMD: —may or may not exist.
CM: First you have to start with the thing. That AI may or may not exist today, is what you’re saying. It’s hypothetical.
MV: So is the particular brick.
ZMD: Again, there’s this idea of subjective probability where there’s a lot of things we think we know about the world. Some of them we’re more confident, some of them we’re less confident. Some things sound really, really wild, and so we’ll assign those a really low probability. But there’s a continuous gradient between certainties. It seems like you’re operating in a mode of thought where this AI thing is in a different magisterium of hypothetical things that don’t actually exist yet, therefore, I shouldn’t use the same kind of subjective probability reasoning about it that I would for a brick.
CM: You’re reading into what I’m saying. I’m just asking the question. You know, I could live my life by saying I’m never going to leave my house because a brick might fall.
MV: But it wouldn’t be rational to.
CM: So why is it rational to live your whole life trying to prevent AI killing us all, even if the probability is low?
ZMD: We don’t think the probability is low.
MV: Nobody thinks the probability is low. That’s the title of the book. The probability is essentially certainty. But also, who’s talking about living your whole life? People do other things in their life. Nobody doubts that even Eliezer does other things in his life. He writes Harry Potter fanfic, for God’s sake.
JT: Someone could also think that it’s so unlikely we can do anything about it, but that’s still thinking it’s a high probability. Someone could think the probability is so high that there’s actually nothing worth doing about it. That is possible.
MV: But the living your whole life thing is weird. That’s just making stuff up.
CM: Well, one of the things I think about is that, basically you said it, what is being said here is the same thing that was being said back in 2008. It’s the same thinking applied to each technology. Maybe we’re really, really far and we’re applying this type of thinking to something that’s really, really far away.
MV: Yes, that’s what we think.
CM: We assign all these, we anthropomorphize this thing that is not—
[...]
JT: I think [trying to see what kind of agency LLMs and reinforcement learning have] is overall a more productive approach than taking this very general model and just assuming that the technology has it. It should be an open question. It’s an empirical thing. I think the stronger argument is that, hypothetically, like future AI algorithms. We don’t know what those will be, right? But we can imagine, you know, people do come up with things and they test them, and some of them work better than others. So we can assume that there’s some selection that people will be more likely to use the techniques that create AIs that have larger effects as a general rule. And an AI that has sufficiently large effects would take over the universe and stuff. I would say it’s just very important to distinguish these things, because we can talk about these specific technologies, like LLMs or RLVR for which, first of all, cat’s out of the bag, you can’t really do anything about it, and second, you can just go empirically study them, versus AI could have any number of future innovations in the future that we just don’t know about.
MV: Okay, so we’ve got a fair amount recorded now, and I feel like one thing that we could do right now is talk to the AI that is going to read the recording, and talk to each other and see what everyone says they think the AI is going to say. Because right now, from my perspective, the central obvious fact is that when you tell your story about a brick could fall on your head, you know already how to use predictive processing, like the AI’s predictive processing, to predict what someone can say in response to that story. You already know the argument you’re making, and you know the counterarguments, and you know the counter-counterarguments. You already know that whole chain. And yet you’re doing it anyway, which is a really weird thing to do from the perspective that assumes that objectivity is being sought from the first place. From the perspective that assumes that objectivity is being sought in the first place, the great mystery, the really important question is not AI. The really important question is, what the hell is the non-objectivity-assuming perspective doing and how do we keep it from killing us? The AI is hypothetical. It doesn’t exist yet. You are real. I know that you might kill me.
[laughter]
CM: My non-objective perspective might kill you, is what you’re saying. I like that. I like that a lot.
MV: I mean, I experience it as an attempt to.
CM: Okay. I liked it. We’re making progress. [to Taylor] Let me ask you a question I asked them. They met in 2008 as Overcoming Bias was coming together. Eliezer’s writing these essays about rational thinking. But as you can see from Zack’s diaries at the time, what they’re really thinking about is, is AI going to destroy the world?
JT: Yeah.
CM: Why is that the natural outcome of this rational thinking that is being laid out?
JT: Okay, so I think that’s a good question, and I don’t know the answer.
MV: I mean, that’s backwards.
CM: Hold on—
MV: No, straightforwardly, though, people were thinking about AI. It wasn’t the outcome. You said, why is AI going to destroy the world, the natural outcome of rational thinking?
CM: I’m asking her what she thinks.
MV: But you inverted the cause and effect. You said—
CM: As you said, I’m trying to kill you all.
[laughter]
CM: I apologize—
MV: Okay, sure.
CM: I’m flawed.
JT: I don’t want to assume a causal order here, right? I read AI: A Modern Approach in high school, right? One thing it describes is Bayesian thinking. It also describes vNM rationality. So people were thinking about stuff like reinforcement learning and von Neumann–Morgenstern utility maximization in the context of AI, and people were thinking about it at RAND Corp. before it was much applied to AI, too. There’s already a pre-existing correlation where if someone’s in AI, they’re probably already thinking about this stuff. If someone is thinking about this stuff, they can take that methodology and then notice that AI is important. I think other things that could be going on is, at some point if you’re thinking about decision theory and rational thinking a bunch, then it kind of opens a question of, well, what if something were better than me at rational thinking or decision theory? And then you’re like, oh, wow, that would be really weird. So that’s one way.
CM: It’s maybe simpler than that, meaning—the other thing Zack and I have talked about is that there’s this moment on the SL4 mailing list where Yudkowsky says, you know, I don’t have the people I need to build this Friendly AI. I need to create this community. I need to write. Every movement needs a book. We don’t have our book. He said, I have to do a book on rational thinking. And a few years pass and he basically does that, right? With Overcoming Bias, he puts together this book that attracts people, and the book that he writes isn’t about existential risk. It’s not about AI destroying the world. It’s about this rational thinking. But as this community comes together, they’re already thinking about that. And Zack talks about, you know, when Less Wrong was formed, there was a ban on discussing existential risk, but that was lifted and everybody started discussing it. Is it because they were already following Yudkowsky?
ZMD: I think so.
JT: I think that was it. But it’s kind of a nice strategy if you think about it, right? Because he’s like, first of all, the thing he notices is that, whenever he’s trying to talk about AI with people, they always make errors, which you would think of as cognitive biases or something. People are just making, or just lack of good epistemology, because he’s making his arguments and people are like, but what about this? What about this? And he’s like, these are all kind of stupid. I want at least 20 people I can talk to who are not completely stupid about these things. And so he writes the thing that is trying to make people not completely stupid about this so you can have a more reasonable discussion with them on AI risk. And the idea of banning it at the start actually makes some sense from that angle, because what it shows is that it makes it look less like he’s propagandizing his particular views. It makes it look more like he’s trying to upgrade people’s thinking, and he thinks that will naturally cause them to believe him about the important things and also just generally have more correct beliefs. It’s not like he’s trying to specifically lead with the thing which is AI risk, which is maybe somewhat more controversial. He’s trying to start with something that is just like, oh, we should think better, which a lot of people would agree with.
ZMD: And the sad thing is that that era is now over.
JT: It’s worse now than it was even in 2016. The problem now is that there is a political movement known as Pause AI or Stop AI and a lot of people are jumping on board with that, and they are saying, we should promote this book, and cheerlead for these things, and they are saying, like, we have no hope except stop AI. This has been kind of an issue, and you’ve noticed it on Less Wrong recently, right? Of the way that people are talking about this book. Instead of just being like, hey, we should just try to have correct beliefs, including about this kind of thing.
CM: So, you’re saying that these beliefs are not necessarily correct?
JT: Not everything in the book is right, to be clear. Not everything in the book is right.
MV: Very obviously. But the point is that it is a rejection of the objective stance to think that we might think that the beliefs were necessarily correct.
CM: Hold on. Let’s backtrack a little bit. Some people would argue that Yudkowsky was proselytizing his own beliefs. You said, this would make it look like he’s not. What if he was? What if it is just proselytizing his own beliefs, and not the absolute truth.
MV: Wait, what do you mean again by absolute truth? As far as I can tell, the absolute truth thing is a pure strawman. It is a phrase that has no meaning at all from a perspective that does not reject it. There’s a perspective that has truth, and there is a perspective that rejects truth and invents the strawman “absolute truth”, a thing that nobody who believed in truth has ever believed in, or ever hinted they believed in.
CM: Let’s just call it truth.
ZMD: But in a previous conversation, I had mentioned, yeah, I have a lot of disagreements with Yudkowsky these days, but the existential risk still seems real because I can check, I’m not solely operating on deference, I can check that part of the argument. And you said, so you still believe in this absolute truth.
MV: And it’s not as if Eliezer accumulated deference through anything but by making arguments that people could check. You know?
ZMD: Yeah.
MV: Self-evidently the deference is drawn from the fact of the checkable argument.
ZMD: Well, not these days.
MV: No, I mean, it is drawn from the momentum of that fact.
ZMD: Yeah. Yeah, yeah. [to Metz] And, like, what did you mean by that question: “so you still believe in this absolute truth”? I mean, yeah, I do think that there is a reality out there that I can try to improve my probabilistic predictions about. What were you even asking?
CM: Well, it’s fascinating to me that this person, who in essence taught you to think this way, taught you that there was this truth.
MV: No, nobody ever told anyone that there was truth. Every infant knows.
CM: But there are a lot of people who would argue differently.
MV: But no children. The point is that there are children and there are liars. Lots of people would argue differently, but obviously they are lying.
CM: [to Davis] I think that one of the [inaudible] things about your story is that you see Yudkowsky denying this truth.
MV: No, nobody’s denying truth.
CM: I’m asking Zack. The way I see it is you see him denying the truth. Does that hurt your faith in everything that’s happened over the past—
ZMD: I mean, can you be more specific about denying this truth?
CM: That’s what you’ve said to me. He’s now behaving like The New York Times where this is not objective.
ZMD: But like, so like … one moment.
CM: Take your time.
ZMD: More than affirming or denying some canonical truth, what matters is processes that converge on truth. One thing that matters is if, you know, you wrote something in 2009 that implies one thing, and you write something in 2016 that implies another thing, there should be a way for someone to say, “hey there’s this apparent contradiction here; how do you account for this?” You can just change your mind. Maybe the new beliefs are actually correct, but you should actually be able to argue, if you believe in the objectivity frame, you should be able to have an account for, actually, I changed my mind because of X, Y, and Z reasons. These days, it seems like a lot of people, including Yudkowsky himself, have this ethos that expecting someone to do that is somehow naïve.
JT: I think he did do this once recently because he was saying, someone called him out on like, hey, it looks like AI is hovering around human level for a while. It doesn’t look like it just zooms past village idiot to Einstein, which you said were very close. And he was like, well, that’s actually correct. I was just wrong about that. And I was like, something, something, youthful idealism. I was underestimating intelligence differences between humans. That was nice to see.
CM: Well, you know, one of the other things that I think sparked this conversation was whether or not this community was like a religion or had religious aspects. One of the things I struggle with is that so many people in the community today, in the past, you know, you see it in your diary, sort of make the analogy. But then people get very upset if I write a story where someone outside the community makes the analogy. The analogy’s there. Can you see the similarities?
ZMD: Yeah, I can see the analogy, but it’s like—so, again, I just …
CM: Where does the analogy break down? How about that?
JT: Okay, so you can compare it to Mormonism, right? Because Mormons have beliefs about space and aliens and the future and transhumanism and stuff, and so do the rationalists, right? But, like, I don’t know. I think with the Mormons, their beliefs are just not credible. Just completely, it’s just very easy to dismiss them. They’re asserting that they are a religious group because they don’t actually want people to just challenge them in academic debates about all this stuff because they don’t actually think these ideas are defensible from generally accepted scientific premises or something. And that is an important difference. If someone is willing to say, the case of reason, even if you don’t accept things on faith, actually does pay for my view, then that actually does make a difference.
CM: So basically the difference is, there are scientific underpinnings to these beliefs.
ZMD: That’s why I said in an earlier discussion, psychologically and sociologically you can compare it to a religion, but there’s more than just the sociological angle. You can actually not just look at the fact that there’s this group and a canonical text that explains why the group believes what it does, you can actually read the text and check whether the text is appealing to reasoning and experimental results, or appealing to divine revelation.
CM: But a lot of people read the text—you’re referring to the Sequences—they’re sort of surprised by it. It doesn’t look like careful reasoning. It’s almost mystical in places. It’s entertaining. It’s interesting. It makes some really interesting intellectual arguments. But also, people debate what it really means. Does it mean this? Does it mean that? You see that even in the comments, right? It’s not necessarily scientific.
ZMD: Okay, the specific aspect of the people in the group debating what the group’s canonical text means, that part is religion-like. I agree with that.
JT: I think it does contain mystical things, right? He is inspired by various Eastern stuff, and he talks about the way of the Void, the virtue of the Void. I think if someone were just trying to write a math textbook or a science textbook, they would probably write something different. I think the thing he writes is somewhat more personal. He’s trying to explain his perspective on a lot of things, not just a single topic. Unlike in this book [If Anyone Builds It, Everyone Dies]. This book is more like a textbook.
ZMD: It’s not really a textbook.
JT: It’s not really a textbook, either.
ZMD: It’s propaganda for the general public and policymakers.
JT: I understand. So that totally applies to Section 3, but I don’t really think it applies to Section 1. I think Section 1 is kind of like pop science or something.
ZMD: Yeah. Yeah.
JT: For example, you can read the sequences, you can get some ideas, or you can read AI: A Modern Approach, which is an AI textbook, and get some subset of the same ideas, and the tone is pretty different. In a lot of them, the tone is pretty similar because they’re explaining essentially the same math concept. But I think Eliezer is trying to make it more humanistic or engaging or personal.
CM: It’s definitely true. The other thing I think a lot about is we have multiple examples where a lot of these beliefs have been taken to extremes, whether it’s the Zizians, Sam Bankman-Fried, Black Lotus. We have a lot of examples of this. What do you make of that? What is it about this community that pushes people to extremes, and can cause problems because of those mistakes?
JT: I’m trying to think of when this applied the most. I think my experience around like 2016 and 2017 was kind of like this, where there were a lot of people talking as if these things are very important, we need high dedication, we need to modify our minds to accomplish this special mission to save the world, and it’s really important. It seems like why would that happen is one question, and I think part of it is that these people think they found something that is really important that most people are ignoring, and therefore people are wrong about a lot of important stuff, and there is something we can do. They believe this more like ten years ago than now. They believe there is something we can do that is almost something only we can do. I even asked Anna or Nate, I think it was either Anna or Nate said, humanity would not have a chance without Eliezer Yudkowsky personally. There was this kind of idea and then there were organizations like MIRI and CfAR founded on this, and there’s also EAs who are essentially utilitarians coming into it. It’s kind of this confluence of factors, and there’s this really important thing, and also consequentialism is approximately true, so you should really try to have a big impact or do something big, and the stuff is really important and other people don’t know about it, and some other people think we’re cranks. I think there’s a way that people who are maybe looking for something really important to be doing can take those things and be like, oh, wow, this is really important. I should dedicate my life to this and I should take actions on the basis of these beliefs that might be pretty bad actions or look really weird if these beliefs were not true.
CM: Why does the community attract people like that?
JT: I think there’s just a dearth of serious things around. Things that actually hold up to some intellectual scrutiny. Clearly religions don’t. There’s various academic things, but they have truth in one area, but they don’t really generalize. People who are looking for something important to be doing, or mission or something, there’s just not that many important things that you could choose. You could choose climate change, I guess. There are people who try to get very energetic about doing something about climate change. That is a thing. But it’s actually less credible than AI risk.
CM: Why is it less credible?
JT: Why is it less credible? So for example, sometimes when people say we only have five years to address climate change, and what they cite is this report about 1.5 degrees Celsius warming. They’re like, we have to act now or else we’ll get 1.5 degrees Celsius warming by 2100. And you actually read the report, and it’s like, if we had 1.5 degrees Celsius warming by 2100, then crops would grow less well, and some people would have to move, and we’d have people moving away from cities. I think one time I just looked into these climate change claims and I went to the report and I was like, this doesn’t seem like a huge deal, honestly. It’s a real problem, but people are saying these catastrophic things, but then you actually look at the details and it’s like, okay, yes, that’ll be an issue, but it’s not a civilizational risk. It’s more of an economic risk.
CM: Whereas AI is a civilizational risk.
ZMD: According to us.
CM: But we don’t know when.
ZMD: Right.
CM: Why do you think that—
ZMD: There is a serious risk of a crying-wolf effect, where people are panicking right now about the AI 2027 stuff—
JT: That is a problem. That is obviously a problem.
ZMD: And then if it turns out that LLMs are not the true superintelligence that Yudkowsky was warning about, then the AI risk movement could lose a lot of credibility—justifiably—for crying wolf, even though the superintelligence risk could still be real like five or ten or fifteen or whatever years later.
CM: What do you make of the 2027?
ZMD: I expect to be alive in 2027.
JT: Yeah, the AI 2027, it’s just very hard for me to take it seriously. Something about the way it’s written or just the ridiculousness of it. I just read it and I don’t even feel like writing a response to it, because it just seems like, what?
CM: Do they believe it?
ZMD: Daniel Kokotajlo has updated his timelines, so now he thinks it’s going to happen in 2029.
CM: Do you think it’s calculated? Like people need to be warned; we’re going to make the timeline short.
ZMD: I hope not.
JT: So I think there is a part of both AI 2027 and this book [If Anyone Builds It, Everyone Dies] that does seem calculated. So if you go through AI 2027, at the end of it there’s two buttons you can press. One is slow down and one is accelerate. And if you press accelerate, predictably the scenario is AI [inaudible] and everyone’s dead. If you press slowdown, it’s like, we only had to delay AI by a few years, and also we all survived, yay, yay.
ZMD: That’s kind of obvious propaganda.
JT: Exactly. And if you read the fine print, it’s like, oh, we conditioned on success. But even if they conditioned on success, their conditioning on success didn’t lead them to longer timelines. I don’t know; it’s just like there’s the thing you could immediately interpret it as, where it’s just like obviously wrong, like why would just a few years slowdown cause this huge effect in outcomes, and then the fine print, even if you look at the fine print and you’re super autistic about it, it’s still a really bad scenario. That just seems like propaganda, and in the third part of this book, some parts also seem like propaganda. They spent a bunch of time being like, this alignment approach won’t work, and this won’t work, and this won’t work; you really need to know what you’re doing; you shouldn’t have false hope in things; this is just cope. And then they start talking about political solutions, and they’re like, oh, humanity still has hope; we can still do something; this is tractable. I don’t buy it.
CM: Why do you think that this community dovetailed with the EA community?
MV: We didn’t. We created the EA community.
CM: Explain that.
MV: EA is just saying that people should be rational about philanthropic decisions. There isn’t anything more to it at all. Rational and objective in the old sense of objective. The whole topic of this whole conversation is, why were you acting within the worldview that everyone had 20 years ago, but which it is forbidden to acknowledge today. Why were you doing that? And the answer is always the same. The worldview that everyone was in straightforwardly implies these things.
CM: Because I’m a New York Times journalist who could potentially kill you, I ask questions. Can you explain to me how you created the EA movement?
MV: Yes, but I don’t see how that’s not a red herring.
CM: I’m interested in red herrings. They’re interesting.
MV: The thing that’s interesting is you, not us.
ZMD: But Michael, so we talked earlier about there’s been a shift in The New York Times since 2008, but Cade’s career has been longer than that.
MV: Surely the shift was over a longer time period, but also surely his behavior has changed a lot over that time period. Surely we could look at his older journalism and see different attitudes.
CM: My point is, you know, people will go to a chatbot, and the chatbot will say EA was created by Will MacAskill and Toby Ord, but you’re saying no.
MV: No! Toby Ord is literally at the Future of Humanity Institute. The Future of the Humanity Institute is the most mainstream, most established, in some sense, thing that is focused on big picture and long-term stuff at all, and is, as anything that is focused in that way, focused primarily on existential risk from AI. And that was always the case from its inception, because that’s what happens when you try to think about the mainstream big picture at all.
CM: Here’s where you and I really agree. I’m completely with you on that. I do get it. But you talk to Toby Ord and he denies it. And you talk to—
MV: Wait, what does he deny?
CM: I don’t want to betray any confidences, but everybody across the community denies they’re part of the community. People deny they’re EAs. If they are EAs, they deny that they have any relationship to the rationalists. This goes on all the time. You know that. Why do people do that?
MV: So, Toby Ord and the Future of Humanity Institute predate the rationalist effort. They are the last effort, you might say, to be a mainstream, big picture, objectively focused institution at all. And when they find themselves in the situation they’re in, they are besieged and compressed and timid. The rationalist community is the people who are responding to their message without trying to be an mainstream institution so they don’t need to be besieged and timid. But in a sense, there’s not much Eliezer is saying, that Bostrom wasn’t saying three or four years earlier, except that Bostrom is not taking the attitude that if you don’t get this, you need to just learn to think better. He is taking the attitude of, I’m going to be a mainstream academic and win these ideas over through legitimacy rather than demanding that people think them through themselves. Eliezer is not claiming legitimacy. He’s claiming that you don’t need legitimacy; this is obvious.
CM: Is rational thinking necessarily utilitarian?
MV: No. Okay, one way of saying this: the word rational, and the word utilitarian, and the word objective, and any words that are in the cloud around rational thinking, are almost necessarily unconscious reactions to anti-rational thinking. Someone who is not being fucked with by anti-intellectual processes would never invent the idea of rationality. They would just behave rationally, expect other people to behave rationally, see other people were behaving rationally. This wouldn’t be an anomaly, and they’d mostly be focused on where they were going to get their next meal.
CM: Is it fair to say that the rationalist community is predisposed to be utilitarian?
MV: No, I would say that utilitarianism is one of many, many, many, many ideas that people create in response to irrationalism, but the idea of utilitarianism has no explanation for its origination except in terms of irrationalism. It’s only in the presence of people who are making moralistic claims that don’t make sense, that you invent a theory of how to make moralistic claims that make sense.
JT: Is this like the abundance movement?
MV: Yeah!
JT: Why do you need a word for, like, we would like better economy? Wasn’t that the baseline?
MV: The point is that there’s a whole slew of thinking that can only exist as a reaction to reactions against naïve thinking.
CM: One of the things in particular I’m fascinated by is, you know, Dario Amodei, he was like the 43rd person to sign the Giving What We Can pledge. And now publicly he says, I’m not an EA. I don’t know what you’re talking about. I’m not an effective altruist. Why is that?
MV: Okay, so that’s something that the whole left in common, and from my perspective, EA is still part of the left. The whole left is, I’m not Antifa, I’m not a Marxist. It’s all based on plausible deniability. I’m not CIA. I’m going to count the far right as part of the left for these purposes, too, because the far right is just the losers who the far left didn’t let join the club. You have the center right, who don’t know what’s going on with power games, and you have the rest of the political spectrum that’s postmodern.
[...]
MV: So like, the Magnificent Seven are basically all of the economic growth in the world. You have the position as the guy in the newspaper with jurisdiction essentially over the story about the Magnificent Seven, from which all of the economic growth of the world derives. You have some books, but you haven’t been successful in a way that would cause you to expect to be in so canonical a position. Like Nick Bostrom, you are in a position that would make you one of the major figures in history if it had happened a century earlier. You find yourself in a position that is still the only official position in the entire world, telling the main story of the entire world that the world tells in itself and that is reflected in the economic statistics, and yet it’s a periphery. So what I’m asking is, where is the center if that’s the periphery?
CM: Where do you think it is?
MV: In the negative space, in the processes of RLHF-like reinforcement that you receive on the job about what sorts of things are going to get published and what sorts of things are not.
CM: I see what you’re saying.
MV: You know, there is literally only one mainstream. Even 20 years ago, there was massive gaslighting about that. Even 20 years ago, it was possible for people to be sort of—very confused about whether we were some hyper niche interest, like polo fans, or whether we were the only line of continuation of the main story. Now the results are in, and we were clearly the only line of continuation of the mainstream story, but the mainstream story had clearly already attenuated its connection to reality by an order of magnitude, and its connection to seriousness or credibility or legitimacy by two orders of magnitude. Elon Musk is very very unambiguously part of the mainstream story and real, but there’s something that will cause people to regard it as not serious. There is a withdrawal of legitimacy from story itself, which is what postmodernism literally means.
CM: Do you have other things you wanted to discuss?
ZMD: I did want to mention, there was something I really admired in Genius Makers, not in the text itself, but on the acknowledgements page. You wrote about a literary agent who gave you feedback, quote, “after he read the proposal I had written, he told me very politely that it was garbage,” end quote. Just that whole ethos that someone can be doing you a favor by telling you that your ideas are bad, is at the core of rationality, of seeking out the best ideas rather than trying to protect your current beliefs and plans.
In that respect, I would say you are more rational than Oliver Habryka and the moderators of lesswrong.com, who just banned one of their sharpest commenters, not even for politely telling people that their ideas are garbage, but just asking questions in a way that Habryka says that he, quote, “cannot help but read in a sneering voice, dripping with judgment, pointing a finger at me or the author in a way that summons judgment and punishment”, end quote.
CM: Who did he ban?
ZMD: Said Achmiz.
JT: Did he also ban TAG?
ZMD: No.
JT: Oh, weird.
ZMD: From your acknowledgments page, it seems like you at least have the concept that when someone tells you your book proposal is bad, it’s not summoning judgment and punishment. They just mean it’s a bad book idea and you should maybe write a different book. The sanity and maturity to recognize that is apparently a large amount of rationality in today’s world. I think the thing that Michael was trying to do in 2009 was to create more sane and rational people, rather than the doomsday cult it turned into. Separately from the fact that doomsday is still real, sane and mature people would be helpful for confronting the impending doom.
The reason the reason I put in so much effort, talking with you multiple times, sending you that 2000 word email, is because I’m worried you’re going to tell the story of this book as strictly the sociological lens of, here are these weird people forming a doomsday cult and they’re pulling the strings in Silicon Valley.
That part is true. The sociological lens is valid. But there’s also this other part of the story, as I said in the email, the reason Yudkowsky succeeded and Kurzweil didn’t, is because before we gave up, before we gave up on objectivity like the rest of the world, except for Michael—
CM: You’re saying that Yudkowsky gave up.
ZMD: Everyone gave up! There was this core insight about how to think better.
CM: We’re closer than it might seem. I want to think better. I want the world to think better. I believe in the truth. I believe in objectivity.
JT: This is a weird thing. Whenever someone asks me whether I’m a rationalist, it just seems like an extremely wrong question. The thing of wanting to be right about things is not restricted to this specific community.
CM: Everybody should want to be right. Everybody should want to be rational. Children are.
ZMD: Again, I mentioned in our August 12th meeting, people who had been supportive of my work were critical of me for the fact of talking with you, because in the Craft and the Community sequence, Yudkowsky had written about why our kind can’t cooperate, and the complaint was, even talking to the guy who doxxed Slate Star Codex is copper-bottomed why our kind can’t cooperate. And it’s just, I don’t believe in that kind of cooperation anymore.
In talking to you, I have this complicated multi-dimensional objective, where on the one hand, I don’t want you to slander my friends, but it’s not because—I hope—it’s not because I’m just bullshitting to cover up their reputation, because I think these people are terrible. They deserve criticism, but they deserve criticism for the right reasons. I was disappointed with the Lighthaven piece on August 4th or whatever, because the whole “This is a religion” angle was not—as I’ve said, there is an analogy, but there’s so much more.
CM: Again, I think we’re closer than you think.
ZMD: Well, I hope that actually shows up in the book, because it sure as hell hasn’t been showing up in your New York Times articles.
CM: The book is different.
ZMD: I hope it’s different.
CM: Believe me, your point is a very good one. Why did it become a doomsday cult? I really appreciate and admire all of you actually talking to me because a day doesn’t go by where I don’t see a comment, or I get an email, or somebody says, you know, I’m not talking to you. That is cult-like behavior.
MV: No, it’s not. That is behavior in response to open oppression. Open social violence of precisely the sort that the people who do it, also spend all their time educating the public about.
CM: Tell me what you mean.
MV: The people who talk about stochastic violence are the people committing all the stochastic violence. The people who talk about structural racism are the people committing all the structural racism. They’re the experts. That’s why they’re qualified to talk about it. You know, White Fragility is an extremely expert, clear explanation of how you ought to mistreat black people in order to get promoted in these structurally racist organizations. And the organizations that hire Robin D’Angelo are saying, we are structurally racist. Here you are as a consultant on structural racism to inform our people. This is text. This is not subtext.
The thing I just said is easy for an AI to understand. It’s also easy for a human to understand if they are not elite. Modus ponens is a prerequisite for understanding as such, and is therefore easy for people to understand if they are not privileged, if they’re not being taken care of for withholding understanding as such.
CM: Let me go back to my original question. Why did it become a doomsday cult?
MV: No! You don’t get to do that!
ZMD: He is using my words.
CM: I’m just using his words. He makes a good point. It became one, but really it missed the point.
ZMD: But also, the thing I’m terrified of, because you’ve totally been doing this in your New York Times articles, and I’m hoping that I’m not an idiot for continuing to talk to you, because I’m hoping the book will be different. There’s this important part where doomsday is still real. The basic obvious-to-a-child case of AI risk, I still believe in that part.
MV: You reveal that you know why it’s a doomsday cult through the things you say, and that the actual answer is unspeakable, is taboo, so only putting a false answer in other people’s mouths could give you a publishable piece. If you didn’t know why people end up in doomsday cults, you wouldn’t have ended up in the type of cult you ended up in.
CM: My cult is what, The New York Times?
MV: Well, the establishment, more broadly. The thing that I’m saying is, structurally, in order for anyone to be informed about anything, one needs a picture of what someone is, what it is to be anyone, and in the natural commonsensical understanding of what it is to be anyone, it’s very confusing what cults even are. In the perspective from inside of some sort of a cult, it’s very confusing what not-cults even are. Any true answer, any even slightly useful answer to why anyone is in a cult has to be an answer to what it looks like, what the world looks like, what it is to be in a cult, interpretable from not in a cult. For an answer to what it is to not be in a cult, interpretable from in a cult. And rather than cult, we could use words like high-context window and low-context window, or predictive processing model versus thinking model. We can use fairly good analogies from the machines that we’ve got now, that can describe the nature of the states, which is a separate question from describing the process of transitioning between them.
CM: I’m going to have to go here pretty soon, but let me ask you again. Why do you think it became, as you called it, a doomsday cult?
ZMD: I mean, again, there’s a lot of qualifiers there, but I think from the full context of our many hours of conversation, you understand the qualifiers.
MV: But wait, the whole point of his agenda is to discard them all anyway, and you know that.
ZMD: Yeah … hm.
MV: And the critical point is that that is straightforwardly lying from the normal objectivity perspective, but it’s not straightforwardly lying from the perspective you’re coming from. So the question becomes, what the hell fucking would constitute lying from the perspective you’re coming from, if the thing that you do every single time you write anything isn’t a lie?
ZMD: Michael, I guess the interesting question is, when he says the book will be different, why am I acting as if I believe him?
MV: Right, that is an interesting question. It has something to do with not knowing how to act as if you disbelieve him in the proper way.
ZMD: Yeah, because I have this faith. I’m going to use the word faith. I think it’s the right word. I have this faith in Speech. Faith that if I honestly try to describe the world as I see it, then maybe someone somewhere will be able to incorporate that information into their map and use that map to steer the world and make good things happen. I’m doing the quokka policy because I don’t know what else I can do.
CM: But you also know that it’s not going to be a book solely from your point of view.
ZMD: Yeah, I know that.
CM: I’ve spent years talking to—
MV: But that has nothing to do with anything. This is just another strawman. The standard toolkit you use for dismissing calls for objectivity is to equate objectivity with blind idiot faith of types that no one generates authentically and that you could never learn to tie your shoelaces if you were acting from them.
ZMD: I understand that the book is not going to be a puff piece. I understand that you’re interviewing and talking to lots of people. I don’t think I’m asking for a puff piece. My faith in discourse is not that you’re going to write the book from my perspective—obviously, that’s crazy—but that you can at least, to the extent that I am a character in the book, you can at least not lie about what my perspective is.
CM: That’s exactly right.
MV: But you don’t do that as a journalist in your day to day articles, and we know that. Have you heard of The Fort Bragg Conspiracy?
CM: Tell me about it.
MV: It’s a book, also by a New York Times journalist, about a very high rate of murder, suicide, and other offenses by special forces. It’s interesting not because it’s fun to read or the topic is important, but because it is journalism in the objectivity sense. The journalist’s attitude is that they are investigating crimes and that in order to investigate crimes, they must make sense of motives, not sociological motives, but motives from a perspective that assumes rational actions by the participants. I can see that in the case of politically-charged investigations of the military by the somewhat more far left of the center, it is still possible to invoke the idea of projecting upon people rational-type motivations and deception and concealment and interpreting. The fundamental challenge that we failed at in a rationalist movement is preserving a context of ability to do that for purposes that are not purely political.
CM: Honestly, I appreciate all of you doing this as usual with me.
ZMD: I did have one more question here. On September 2nd, you emailed me saying you were doing a profile on Yudkowsky for the Times and asking if I could help fact check. And then they gave the story to Kevin Roose. Do you know what happened there?
CM: No comment.
ZMD: Alright. Because I had a theory that Yudkowsky was willing to talk to him and not willing to talk to you, and so that’s why they gave the story to him.
CM: No comment.
ZMD: Alright.
CM: I’m glad you asked.
ZMD: I was just curious.
CM: Yeah, yeah. Maybe one day we’ll talk about it. Thank you. I’ll keep you updated on how this, ah—
ZMD: Do we know when the book is coming out?
CM: Not yet.
ZMD: I mean, I’m not optimistic, but I don’t think I’ll regret my faith in Speech. I don’t think it’s going to be a good book, but it’s not my fault.
CM: There you go.
What the hell is this. Feels like four people all trying not to have a conversation in four different ways.
ZMD: And i just, i’d like to tie that back to the way that less wrong moderators banned Said. I think you need to verbally acknowledge that.
CM: Got it.
ZMD: You keep saying that. You keep saying that while giving no indication that you’ve understood anything we’ve said.
CM: Right, right. I think we’re closer on this, we’re close on this.
MV: Zack, if I may. [to Cade] We are not the topic here. You are the topic. The far-left post-structural gestalt is the topic. This is not subtext. This is LLM text.
JT: Sorry I’m late --
ZMD: It’s ok. We were just discussing how recent moderator action has turned less wrong into a doomsday cult. Habryka needs to ~~bake an apology cake~~ promise to do better in the future.
JT: -- I was watching Ex Machina.
CM: “doomsday cult” [sic] more say more about that say “doomsday cult” again use that phrase again please on the record “doomsday cult” for the book the title say it
MV: Any child can see these things. You can’t explain them. You can’t explain anything to children.
JT: This has become political.
ZMD: No, it’s fair. I did say “doomsday cult”.
MV: All cults are ‘doomsday’. The whole concept of ‘doomsday’ only makes sense as a response to a response to a common knowledge misunderstanding of wrongheadedness. If you were walking around and there were only ordinary cults, you would never have to use the word ‘doomsday’. You would never invent that word. [to Cade] You must have seen this from within your own cult. You are categorically unable to understand anything I’m saying. You reject truth. How do you respond to these accusations?
CM: [nods] Got it.
(fondly!)
touché
It mostly makes sense to me? I think?
It locally makes sense to me, what people are trying to do, and why they say the things they’re saying given what they’re trying to do. But not why they’re trying to do what they’re trying to do.
Oh ok. That’s a pretty high standard (I mean, a reasonable standard, but also high).
The first rule in talking to Cade Metz is do not talk to Cade Metz.
And that’s phrased as a joke, but I mean it seriously. He is out to harm us. Please stop helping him do so. Quokkas usually just don’t understand when someone is going to harm them, but in this case you don’t understand he is out to harm everyone here.
(Also, linking this comment may make this point more explicit.)
IMO there is a huge unilateralist curse here. But it’s worse because the people who want to talk to Cade Metz are not just more optimistic than average, but worse at it than average.
Kelsey is probably good, but Michael Vassar is nearly pessimal for the job: an actual manipulative cult leader type that Metz can see past.
I don’t think Metz can “see past” Michael Vassar. I think Metz is executing an adaptive strategy which boils down to do not engage with this individual. Metz is not modelling what Vassar believes in any detail; he is essentially just nodding along. This is a locally smart strategy given what happens to (some subset of) people who do engage with Vassar’s ideas, but it’s a rock-like strategy rather than an intelligent one.
As I mentioned in a previous interview, I think these labels get thrown around as insults without being based on anything substantive:
Changed “cult leader” to “cult leader type”. I think the fact that he was manipulative and caused negative outcomes in several people
includingdrug-induced psychosisis well-established.No, it isn’t: see Scott Alexander’s followup comment in which he reports that he “interviewed many of the people involved” and that his “main conclusion is that I was wrong about Michael making people psychotic.”
situation considered complicated, calling it pure selection effect is cope at best, enabling at worst. calling it pure causality would be a ridiculous witch hunt. truth is messier and counterfactuals are hard.
Thanks for the correction, I try to avoid drama and sometimes it means I miss correction comments and such.
I’m still wary of the process but don’t really want to engage more
What is this “us” that will be harmed by Zack talking to this guy? Game this out for me. Who is harmed, in what specific way, by Zack giving interviews like this?
I liked Jiro’s littering analogy. I assume the causal story goes, rats talk to Metz → Metz has an easier time writing articles and a book portraying rats as unreasonable/low-status (which don’t have to rise to the level of overtly hostile hit pieces) → people who trust New York Times reporters have lower opinions of rats → those people e.g. discriminate against us in hiring or support worse AI policies in expectation. I buy that there’s a real effect here; it just seems small enough that I don’t particularly feel like suppressing my natural inclination to talk to everyone and do cool meta-journalism.
Rats talk to Metz → Metz has an easier time writing about rats → more people learn about rats → there are more rats → rats talk to Metz
My issue with this is, you can’t stop someone from writing a book. Did you think a professional journalist is going to shrug and give up if the group they’re profiling successfully coordinates not to talk to them? If I didn’t volunteer to be the star cooperative source, the book just gets written with David Gerard in that role, which is even worse.
And if David Gerard decided that Metz was unethical enough that even he wouldn’t talk to him. Metz could just invent a conversation with rationalists from scratch.
I don’t have an exact link to the article right now, but Scott once pointed out that even when people lie, the lies contain true elements when they could have just as well made up a complete fabrication. Refusing to talk with Metz increases the amount by which he has to lie, and makes his lies less convincing when someone catches him doing it, even if lying is still possible. If all he has to do is pick out a juicy quote which comes directly from a rationalist and just has to have the preceding sentence removed, you’ve made his tactics a lot easier than if he has to say something like “I spoke with David Gerard, a prominent critic of the so-called rationalists, who confirmed that...”, even if in theory you haven’t stopped him.
I also find this to be a case of a general rationalist flaw: refusing to take responsibility for making bad things easier on the grounds that they could have done it anyway, when in practice making it easier to do something that could have been done anyway drastically affects people’s behavior even if that seems to be illogical. Getting rid of trivial inconveniences matters.
(Also, if he talked with David Gerard specifically, there would be a lower chance of getting the lies into Wikipedia, since Gerard’s conflict of interest is known there.)
No, he couldn’t. Journalists who outright make stuff up (as contrasted to engaging in mere “spin”) get fired in disgrace.
That’s the point. Metz could still lie if even Gerard wouldn’t talk to him, but not all lies are equally easy to get away with. Making his goal harder matters, even if strictly speaking he still has ways to reach it.
Okay, I think we both understand the problem with selective reporting. (An adversary who wants to minimize an audience’s estimate of the Heads-bias parameter of a biased coin can report only the Tails outcomes, and if the audience trusts them to be reporting fairly, that’s a huge distortion on what the audience ends up believing.) So this is probably a quantitative disagreement about what Metz’s goals and constraints are, and what my goals should be?
I understand and concede that in the limit of a sufficiently intelligent and sufficiently dishonest reporter with sufficiently many readers who trust him sufficiently much, whose only goal is to maximize reputational damage to their subject, the subject absolutely shouldn’t speak, because any information revealed is going to be perfectly cherry-picked and spun in a maximally damaging way to a maximal audience.
Given that foundation, I think there’s an empirical question of how close Metz is to that infinite limit along each of those dimensions—whether he’s close enough that the act-consequentialist cost of reputational damage to “the community” (which I have common interests with) outweighs my non-act-consequentialist committment to transparency and the benefit of getting to publish cool meta-journalism blog posts.
As I repeatedly express in the transcripts (“I’m hoping that I’m not an idiot for continuing to talk to you”, “why am I acting as if I believe him”, &c.), I have some uncertainty about whether I’m getting the calculus wrong here, but in order to update, I think I’m going to need more details than “he is out to harm everyone here.”
Certainly, he has no qualms about doing harm to people here. (We see that in his reaction to the Slate Star affair, where he displays no remorse for his role in Scott Alexander leaving his job.) But I think there’s an important difference between intended harm and indifference to harm in the course of reporting on a real story (that an ideological internet community is wielding influence in Silicon Valley).
Maybe it would help to be concrete about what harm you foresee? My median expectation here is that a book gets published in 2027; it kind of sucks (similar to how Metz’s previous reporting kind of sucked) and sells maybe 40K copies. The harm dealt to users of this website (in the form of people reading it and treating us worse, or making worse decisions about AI safety) is plausibly less than that of Metz’s August 2025 piece on Lighthaven and the Secular Solstice (which, to be clear, kind of sucked), because more people will read a New York Times article than buy an entire book.
Did your life get noticeably worse as a result of that article (which, as far as I can tell, didn’t even make the print edition)? Will you forever rue the date of 4 August 2025 as the day of an unspeakable injustice and tragedy, a wound in the world that can never be healed? Maybe this seems monstrous to you, but I just find it really hard to care about such diffuse purported harms compared to the immediate benefit of me getting to publish a cool meta-journalism blog post.
Imagine a suitably anthromorphized lion wanting to eat a sheep. Does the lion intend harm, or is the lion merely indifferent to harm? You can argue that the lion just wants dinner. The fact that the sheep has to be killed in order to become dinner just happens to be a step towards the lion’s ultimate goal. If the sheep was harmed in some other fashion, let’s say being thrown into a volcano, the lion would have no desire for that to happen at all! And if the lion managed to eat his fill in some other way, he might not even care about the sheep (until he got hungry again).
So clearly the lion doesn’t actually intend harm to the sheep. It just wants dinner. It’s indifferent to the harm that happens to the sheep in service of its goal.
But if you’re a (suitably anthromorphized) sheep in a place full of lions, you still had better take measures against lions eating you.
In some literal sense, Cade Metz is merely indifferent to the harm he causes to rationalists by lying about them. But he’s still a threat. His terminal goal is incompatible with rationalists’ welfare, even if he wouldn’t throw rationalists into a volcano just because it causes them harm.
The problem with this question is that the exact details of the harm can be wrong in a way that lets you nitpick the answer, but which has no relevance to the question “how should you deal with him?”. My rough guess is that he is a particular kind of leftist who is ideologically hostile to techbros and the tech industry and associates them with fascists, and is going to spin anything you say along those lines whether it is truthful or not. But the exact details of what he’s going to do and how he’s going to do it don’t matter, because harm based on one ideology is pretty similar to harm based on a somewhat different ideology.
Does anyone’s life get worse as a result of throwing one piece of litter on the ground in the park?
Talking to hostile media is a collective action problem.
This is itself something you are messing up. It’s the equivalent of the sheep saying “Nice juicy mutton right here.” You seem to think that being half-hearted about admitting your naivety only causes half as much harm than completely admitting your naivety. It doesn’t work that way; doing it halfway hurts you about as much as doing it fully.
Comparing this to that other Vassar/Metz interview from 2020, it sure doesn’t seem that there’s been any progress whatsoever in the six years since. I would suggest curtailing this project, @Zack_M_Davis.
In a comment elsewhere, you argue:
First, I think your interventions are not going to make the book much better at all. You are not getting through to the guy, and it’s not as if, merely because he talked to you, he’s now going to not talk to Gerard or whoever anyway.[1] And after he’s had all those conversations, he’s going to include only the parts that let him paint whatever narrative he’s already decided is most juicy. Inasmuch as your segments would appear there, it would only be in those capacities that help him in that task.
Second, participation legitimizes it. If nobody from the community were willing to talk to him, he’d have to publish a book which mostly contains perspectives of people unrelated to the community or on its fringes. I. e.: much worse sources, less accurate information, less credibility attached to those sources. It would be much easier to win the information war against that book, to dismiss it as lacking valuable content.[2]
Interacting with the guy just feeds him information for making his narrative more convincing, while likely barely changing what that narrative is going to be. E. g., now he can put quotes in his book in which a bunch of people from the community literally call it a “doomsday cult”, and as was noted, he is not going to include any of the relevant implied caveats.
Third, this is plainly a waste of your time.
I mean, thank you for your sacrifice, the interview was entertaining to read and I appreciate the effort. But I don’t think it’s a good idea to keep doing this.
Indeed, the guy prides himself on looking at things “from all sides”.
Which is the third type of intervention you don’t seem to be considering here. If you want to make the book’s impact less bad, you can try to make it better, or you can try to prevent it from being written, or you can prepare to launch a counteroffensive and dismantle it after it’s published.
I mean, he should talk to Gerard. There’s a lot to dislike about lazy “both sides” journalism where the reporter writing a story about X just quotes a pro-X person and an anti-X person and calls it a day, but you do want to at least talk to some anti-X people as part of the process!
I worry that my other comment that you quote, read in isolation, might have left you or other readers with the impression that my goal is to protect “the community”’s reputation. To clarify, I don’t really care about that. (As suggested in the other comment, I do think it’s plausible that someone whose goal was to protect the community’s reputation would do well to talk to Metz in order to prevent the void of sources from being filled with hostile actors like Gerard, but someone playing that strategy should speak much less candidly than I do.)
I care about public knowledge—the construction of maps that reflect territories. As part of that, if someone is researching a topic I know a lot about, I’m happy to chat with them and take my own recording and publish my own transcript. If the person I was chatting with writes a bad book, I feel like that’s not my fault. If someone else wants to write a better book, I’m happy to chat with them, too, or they’re welcome to draw on the published transcript of my conversation with the worse author. I would never talk to a journalist who wouldn’t agree to me publishing my own recording!
Then you yourself are a hazard to the community. You are helping destroy it by talking to Cade Metz.
One of the many flaws with “rationalist” thinking is that rationalists won’t recognize that whether you make something easier or harder will drastically affect how much you get—even if the final decision to do it is made by someone whom you have no control over at all.
If you build a crypto system that is useful for money laundering, rationalists will say “Oh, I wanted it built for something else, people chose to use it for money laundering. It’s their own choice, I have nothing to do with it.”
If you build a system for prediction markets and people use it for gambling, rationalists will say “it is people’s own choice to use it for gambling. We have nothing to do with it!”
Rationalists will propose a banned products store where you can buy dangerous things and say “if you choose to buy something that kills you there, it’s your own fault”. (Mercifully, at least Eliezer doesn’t believe this one.) See also: rationalist calls for abolishing the FDA.
If you do things that make abuse or misuse of the system much easier, a significant amount of fault is yours for setting up the system that way, even if the person who will ultimately do the deed is someone you don’t control. You are not absolved from responsibility just because it’s someone else’s own choice.
“It’s Metz who will be telling the lies. Poor old me is just committed to truth and is not responsible for what I made it easier for Metz to do” is abdication of responsibility. It absolutely is your fault, even if Metz is also contributing.
The text of the very post you cite is arguing against that stance.
The point is NOT that “If you made your own choices then they’re your own fault”. The point is “there’s an actual cost of people who realistically will make these mistakes, who will be harmed, and we shouldn’t pretend otherwise, or slip into justifying it.”
What? I specifically stated that Eliezer didn’t buy into it, which is what you are quoting.
Systemic actions and personal actions are different. The analogies you give are all systemic:
crypto system
prediction market
banned products store
things that make abuse or misuse of the system much easier
setting up the system
But the action you are critiquing is not systemic, it is talking to a journalist. Systemic analogies are misleading. Instead I’m thinking about analogous actions where someone takes a small individual action that indirectly negatively impacts a group’s reputation, but only because of the poor choices of other people. Hmm.
How about this very comment I’m making? Davis’s essay discusses the New York Times. By replying to this comment I’m making the essay slightly more visible. Other people may read it and have a worse opinion of the New York Times. Some of them will make unreasonably large updates. So now, am I a hazard to the New York Times? Am I helping Davis to destroy it? Are you? Nah. When people read stuff I didn’t write and think things I don’t think, that’s their problem, not mine.
(and what if someone reads your comment and has a lower opinion of rationalists as a result?)
See Against Responsibility for more on where I’m coming from. Related is Asymmetric Justice—it’s especially pernicious if Davis is blamed for Metz telling lies but doesn’t get credit for inspiring me to be more honest.
Eh, fair enough, thanks. I have to admit you have a point: to some quantitative extent, I’m a hazard to the community’s reputation. If the community would be destroyed if it had a sufficiently bad reputation, then I’m helping destroy it (to some quantitative extent) by speaking candidly with someone who will predictably use the information they gain to lower the community’s reputation.
(I’m saying “to some quantitative extent” because, as I explained in another comment, I think the effect sizes here are pretty small, but as you correctly point out in another comment, that doesn’t justify rounding down to zero: someone who litters in a park is making the park less nice, even if any single piece of trash doesn’t ruin the park.)
I don’t care! I just don’t care very much about covering for the community’s reputation! I care about telling the truth in public, which includes doing meta-journalism like this post. I think I care about the community’s reputation a little bit, such that I would engage in some strategic concealment if “destruction” were actually at hand—but on the current margin, no, I’m not going to stop “littering”. What are you going to do about it? Lobby the admins of this website and the organizers of local meetup groups to ban me (a 9x Curated, 4x Best of Less Wrong, 3x Less Online invited author) for insufficient loyalty? Fine! Do your worst! I’m long past the point of being controlled by you people.
This is antisocial. Hurting the community and saying you don’t care is like robbing a bank and saying “I just don’t care much about whether the bank gets to keep its money”.
Refraining from harming others is not supposed to be an optional matter of personal preference.
Zach is adhering to a coherent notion of morality here. I think I agree that what he’s doing is antisocial (according to a relatively common notion of “antisocial”).
But Zach (I believe) disputes that being antisocial in this sense is bad. Indeed my impression is that he thinks the opposite—being prosocial in this way is the active source of a lot of harm and evil in the world.
Right. Telling the bank robber he’s being antisocial doesn’t work, because he already knows that. What you need is a police force to throw the robber in jail. Then people who don’t want to go to jail won’t even try to rob the bank. That was what the meetup ban suggestion was about. If even that fails to deter potential snitches and your internet community can’t escalate any further (because the local non-internet police insist on maintaining their monopoly on violence), then your community might not be able to successfully protect its reputation.
Do you even believe there is such a thing as right and wrong when no police or similar enforcers are involved?
Yes, of course. I’m not a monster—at least, not according to my moral intuition. But sometimes there’s not an obvious morally right or wrong answer when the interests of multiple parties are at stake. In the case of “my interest in having candid conversations with people and doing cool meta-journalism blogging” versus “the community’s interest in protecting its collective reputation”, the latter just does not seem compelling to my moral intuition. If that makes me a monster with respect to your moral intuition, well, that’s when you need the police.
But “I don’t care much about the interests of the other party” is not a necessary part of comparing those interests.
If you don’t care much about the interests of the other party, the outcome of comparing their interests to yours will, of course, be that you always win.
Nah, I know your general disposition towards that. But I did get the impression from the transcript that you’re talking to Metz in an attempt to shift the impact of his book from one you consider more negative to one you consider more positive in some way. I am saying I don’t think this will work, that he’ll only use the information and credibility you lend him to twist his readers’ minds further away from the truth. It would be better if the book were a sad little thing filled with vicious rants from uninformed onlookers, so that nobody could end up mistaken that it’s of any value.
FWIW, I also think the effect sizes here are really small. Indeed, I think you’re not getting to the guy so much that he’s not even going to be able to competently use the information you give him to strike at the community or whatever. If you feel that you’re getting back your time’s worth by being able to make cool meta-journalism posts out of it, I guess knock yourself out – those posts have been entertaining. (Though I’m not sure how much of that you’re going to extract from Metz either; as I’d pointed out, it’s all starting to read very same-y.)
I and a few others publicly state “long” timelines, but I wish someone more prominent in whatever the relevant social contexts are would say so clearly, while also saying IABIED. That way maybe someone would have credibility after this happens. (Though IDK if it makes sense to assume that credibility would change like this; it doesn’t especially seem to do so in a lot of other contexts, but IDK.)
I very much enjoyed reading this. It is deeply weird in a way useful to me.
I would really like to analyse all the conversational techniques Vassar is using here. It’s very interesting and you can see some of the mad charisma he has, and maybe come to understand part of the hold he has on some people.
It’s like someone took a Seinfeld sketch where the gang is holding an intervention for a reporter who’s doing a story on them, but the sketch is written by a rat in 2026.
It’s funny to see Metz going the extra mile for his book, trying to be buddies with his accusers in the rationalist struggle session, all so he can ask his questions for the nth time in hope of a quotable answer…
But more interesting to me would be to finally understand what the “Vassarite” philosophy is. There is the interesting rhetorical move to say there’s a natural and correct way to think, there is semi-deliberate irrationality by the human race, and the only reason we have people making “objectivity” and “rationality” (and even “utilitarianism” and “abundance”) into explicit values, is to reassert natural thought against pervasive unnatural distortion. There is the idea that leftist postmodernism took over American institutions a decade or two ago, and is the biggest and most important example of retreat from natural rationality (with right-wing populism regarded as a further deviant offshoot from its left-wing enemy). Then there’s the account of the rationalist movement itself: Bostrom’s institute was an attempt to talk about the transhuman AI future under the assumption that natural rationality was still hegemonic in the mainstream institutions; Eliezer’s institute (in the Less Wrong era) was an attempt to talk about the transhuman AI future independently of the mainstream, legitimized instead by an explicit doctrine of rationality; but even Eliezer’s organizations have deviated from rationality now, so they can be heard and get along within the now-irrational mainstream, and at this point it’s only the Vassarites who are adhering to rigorous full rationality.
Does it really come off as extra-mile struggle sessiony? I would assume that a New York Times journalist is used to much tougher conversations than this one.
This is the closest this community has come to becoming a religion.
“You knew this when you were a child” is a common argument of Christian apologists; they claim you knew it intuitively and rejected it as a result of social pressure, and they’re trying to lead you back to the truth. (Whether or not it is the truth is a separate question.)
I don’t believe Vassar has asked a child what they thought of us making an AI that destroys the world (he doesn’t feel like that kind of man), and even if he had, that archetype is now abundant, so it would prove nothing.
A lot of obvious things are wrong. “Down” is a different direction in different places, or sometimes not defined at all. Trees are mostly made of air. Lightning is not thrown by a god out of the heavens. Being rational is about throwing those convictions out and taking small but sure steps towards truth (“updating your priors”), not claiming that you have it in your hands and what do you mean, you don’t see it!?
For an attempt to convey how AI risk is real and grounded, far too much of it was spent convincing Cade that something we wanted explained to the public did not have to be explained. Vassar led the conversation into an unrelated and undiplomatic aside on the NYT’s credibility, which was the only time in the session any AI was used and required another aside in order to choose one to the group’s satisfaction. This was not a good use of time or Cade’s goodwill.
Scene: Michel Foucault, having been transported forward in time to see that he has won, at last feels that he will be understood, sits down with a nyt reporter. Foucault slowly realizes that winning precluded not just him getting understood, but the possibility of understanding. Comedic timing, make no mistakes, unless they would be funny.
I don’t know what to say lmao. He is not thinking clearly. This is triggering my “I can fix him” reflex. (I think I’d back up and explain how one can know that playing the lottery is usually a bad idea, and how to recognize the rare cases where it isn’t. He’s not thinking in a quantitative way about uncertainty. Which is understandable, because it is a rather twisty idea — the idea that uncertainty, the absence of knowledge itself, can be quantified and measured out in precise increments. Maybe I’d tell the story of Leibniz thinking rolling an 11 and rolling a 12 are equally likely when rolling a pair of dice; maybe I’d challenge him to think how that mistake could be rather significant if you made decisions on that basis, by analogy to the lottery. Then I’d proffer the notion that phrasing things in terms of gambling and betting is a metaphor; the real idea is broader, about trying to make the best decisions you can under conditions of uncertainty. And he’d object, so I’d tell the Tragedy of the Austrian Economists and their dogmatic commitment to frequentism; and how it’s good the modern-day experts have moved past that sort of thing.) I guess this sort of thing is what compelled Yudkowsky to write the Sequences.
Also, this is my first time reading anything at length from Michael Vassar, after reading lots of people say wildly varying things about him. My main impression is to wonder, why does he talk like that? Does he believe he is communicating clearly, or does he know he isn’t and doesn’t care? He is rather quick on his feet. My guess is that he enjoys the disarming effect that has on people, and he’s making a mistake in letting that enjoyment limit how much he effectively communicates, to the detriment of his other goals.
Metz’s persona preys upon that reflex.
Peripheral to any larger point, but: it’s quite charming that our standards would count that as a slogan.
FWIW I do have nice short useful mental handles for a lot of the things I’ve learned from this community, many of which come from things written in this community, but they make no sense to anyone who can’t unpack them into a version at least as long as that and aren’t useful to those who haven’t done the thinking. Tsuyoku naritai, whale cancer, you get about five words, expecting short inferential distances, double illusion of transparency. Things like that.
Back during covid I once quoted Zvi to a friend who didn’t read his stuff or LW about “pretending to pretend to try to try” and she was thrilled that there was some audience, anywhere, that could read that and nod along because it made sense to them. And it does make sense. You just need a lot of context to see it.
MV here I think explains why. You don’t need slogans to explain things that are obvious to children. And you maybe can’t make short slogans that are both immediately comprehensible and sufficiently accurate counters to the slogans people make to confuse others into disbelieving the things that are obvious to children.
Well, actually, there are a few, but I think they sound a bit silly and childish and condescending unless you know what they’re a response to. For example:
Death is bad.
If the sky is blue, I want to believe the sky is blue, and if the sky is not blue, I want to believe the sky is not blue.
If Hitler says the sun is shining that doesn’t make it dark out.
What is true is already so, and admitting it doesn’t make it worse.
Here’s an old extensive thread about Michael Vassar, that I found to help understand his intellectual style and why some people find him valuable. Figure I’d share it in case others have similar questions.
Trying to use LLM’s to back up your arguments in real time is insane. This seems self-evident to me, but I’ll briefly list a few of the issues.
Hallucinations, bias from the prompt, bias from memory/custom instructions, bias from the specific model, having to read a long response just to see if the answer is a good one, LLMs just sometimes having poor reasoning skills, LLM’s not focusing on the point you were trying to get across, lack of transparency with what you’ve pre-promoted the LLM with, I could go on.
As a layperson MV came across quite unconvincing with his arguments. The children find it obvious is extremely unconvincing as is the list of 3 stories that have the creation do something bad. The segue rants didn’t help the case and mostly derailed the conversation.
One thing I did get out of reading this was an appreciation for how little I understand of how culture/cliques effects how the world is understood. I find the behaviour of much of the world bizarre but I also find the behaviour of these rationalists bizarre. It’s something I need to reflect on but I’m open to explanations.
What about calibrated belief. Consider Laplace’s estimator.
I found this very interesting however
Just wanted to say I found this conversation very interesting - so much so I will be taking a look at the previously linked ones. I think I also found it useful. Although I’ve read what I think is a significant amount of Rationality texts, I still feel that for some very basic stuff that is touched upon here, I am still as flabbergasted as much as any layman, and that the talk helped clarify some things.
Wait, is he saying that he played a role in the writing of IABIED?
No; Vassar and Yudkowsky haven’t been on good terms lately. I would assume he meant that the pattern of non-elites being more receptive to anti-extinction arguments also held back in the early 2010s when Vassar was leading the then-Singularity Institute, and that he thinks this has informed MIRI’s post-2022 public outreach pivot.
This is fascinating. It’s pure uncut power normie vs the most rationalist of rats, and all of them are being earnest and trying their best within their own frames.
Of course, the journalist’s framing means that I bet she’s going to keep looking for the “bad thing foundational to my tribe’s enemies” angle unless another better way to fight her tribal war emerges from the interactions.
And I bet these rats will be betrayed and possibly sacrificed by whatever story she eventually tells. But this feels like real progres over rat political moves of years past—someone show this to Robin Hanson.
That’s true in the sense that no claim about the future is ruled out about facts about the future, and false in the sense that any such claim automatically has any epistemic weight. Anyone can make an unjustified claim , but if it’s only an unjustified claim, there is no reason for anyone else to agree.
It’s a rather vague statement, and whether its right or wrong depends on the exact interpretation.
How does that help? That the territory is not blank just means that something will happen in the future. It doesn’t tell you what it is. It doesn’t offer evidence or justification.
If we can’t determine it, it’s undetermined in that sense, even if it’s ls determined ontologically “in itself”. Which is relevant? Actually , it’s the first sense: what the debate is about is the epistemology, what we can know about the future,because Metz is objectivlng to unjustified claims, mere opinions
How does that help? Yes, in a block universe theory, the future is already there , but that still doesn’t mean it’s knowable.
What helps knowability is laws and patterns, not mere existence.
What’s the difference? If you assume an arbitrary prior, you can literally assert anything. Saying it’s probabilities doesn’t change that,because it is subjective probability (the thing that should properly be called credence) not objective probability the kind of probability that is based on empirical evidence. Bayes allows you to adjust your probabilities by evidence once you have evidence … but you don’t, about the future.
Yes , you can have evidence-based probability about future events where they follow the same pattern as previous events—inductive probability.
The better argument against Metz is the one about based on empirical evidence about geometrical growth in capabilities. Even if we don’t know how to build ASI (assuming it’s not just a series of incremental improvement cements in current LLMs,which we also don’t know), we have data about other technological revolutions. We are not in a position of comple ignorance. Whatever point he is making , there are ways of countering it—which have nothing to do with map/teritory distinctions.
Yes, but they are not knowledge unless they relate to something in the present or past—in which case they are not wholly subjective.
Wholly subjective claims are in no way different to the thing Metz is complaining about. If you want to sell probability to someone, you had better make sure it’s not subjective probability. Certainty is unobtainable but subjective probability is valueless
Probability as opposed to credence actually is. Using the same word for two different things confuses the issue.
I think, early on in the first conversation, the metaphor of territory has more legs and could maybe help explain things. For example, if there’s a town and there’s GeoWizard doing a straight line challenge, he might not know how to get there, but you can fairly confidently predict that if you post up in that town you’re gonna see him roll in some time afternoon fairly reliably. Whereas if you’re awaiting the summiting of Mount Everest there’s an unpleasantly good chance that whoever is going up there is going to come down in a bodybag or not at all. And in fact you don’t actually need to know anything particularly about the terrain, if you just look at a map and this person says “I will get to there or there” and you can see them make steady progress day after day, you’re probably looking at a map of the British countryside instead of the Himalayas. And there might of course still mountains and cliffs and barbed wire fences and military areas, and you may not know how to get there and even GeoWizard may not know how to get there. But the thing about being an experienced expert is you can navigate around obstacles in a reliable, repeatable manner, and this makes the entire story a lot more predictable than one would think.
So the point is, so far it looks like AI development is certainly not easy but on net more on the level of somewhat impassable countryside than alpine range. And maybe the destination will turn out to lie behind a cliff, but maybe not. So if your safety depends on GeoWizard not getting to Bamburgh or whatever, the fact that even he may not know how to get there until he tries it is simply not of any comfort.