I appreciate the effort you’re putting into this thread here. While I haven’t changed my mind on high-level strategy, I have learned at least a couple of things here, so I do want to reassure you that your efforts aren’t just hitting a brick wall here.
That said, let’s take a look at one of your quotes: “Human extinction isn’t that far out of the Overton Window at this point; people like Musk creating big headlines talking about it every year has moved it firmly adjacent to it, if not inside. It’s half of what gets talked about nowadays on some of the most-listened-to platforms in the world, like the Joe Rogan Experience, whose audience is most certainly not an educated one.”
The main question is—how do you think this happened? Human extinction is weird and out there. You yourself don’t believe it. But you know about it, which at least gives you the possibility of agreeing or disagreeing with us. This happened through awareness building. I don’t think this happened as an inevitable result of AI’s ascendancy. I think LessWrong style advocacy around extinction risk was a direct contributor, and the ability to say something like “X-risk is half of what gets talked about nowadays in the Joe Rogan Experience” is what it looks like for our advocacy to be succeeding.
Furthermore, I don’t think we’ve hit steeply diminishing returns to awareness. This post https://www.lesswrong.com/posts/A7BtBD9BAfK2kKSEr/what-we-learned-from-briefing-140-lawmakers-on-the-threat is pretty enlightening here—it’s not too out of date (Sep 2024 - Feb 2026) and describes that most British MP’s are unaware of basic facts that support our argument. I think a lot of people who disagree with our argument do so not because they’ve developed detailed models of how we view the situation and they think we’re wrong on certain key points. I think they disagree because a sensible prior is “This technology will not kill everyone” and they haven’t heard enough information to make them consciously reconsider this. There is actually key information that is required for the extinction argument to make sense that many people don’t know. I don’t think our arguments have failed to connect with all such people. I think they’ve often simply never been made in person with sufficient fidelity to drive our point home.
Note that I am not accusing you personally of this. It is entirely possible to understand our argument well and still disagree with it, and some such people exist, and may be basically unreachable as you say. But I think this does not yet account for most of the important people who disagree with us.
There’s also common knowledge—it takes an unusually courageous person to stand up first. Asking someone “Please stand up against extinction risk, even though you may be alone and the confused silence will be deafening” is a much harder ask than “Please join our large and growing group of people who are openly concerned about extinction risk and lend your voice to the crowd” and we appear to be moving rather rapidly towards the latter. Anyone who is convinced by the second but not the first is very far from unreachable.
I think your strategy has non-zero value, in the sense that your strategy is better than doing nothing. But I don’t think it’s better than what we’re doing. I think that:
We have successfully changed extinction risk from AI from “A thing nobody has heard about” to “A thing that is discussed in halls of power and major media”, and this is a big change.
There is still a lot of utility left in educating people—reaching new people and making the case to people who have vaguely heard about extinction risk. Most people who disagree with us don’t deeply disagree with us to the point of being unreachable.
The best way to educate people about extinction risk is to talk directly about extinction risk.
This has higher value than attempting to use existing prosaic anti-AI sentiment to push obstructionist regulation. Although I have been convinced this is a good (i.e +EV) strategy, I don’t think it is the best strategy.
Thanks for the acknowledgement that I’m not hitting a brick wall! I’d happily die on any hill,[1] but it’s always more enjoyable when dying on it to open ears.
The reason I was exposed to arguments around x-risk is the same reason that I’d imagine most people who have been actively making the “problem” worse know about it: It’s something that a fanfiction author who was unusually effective at popularizing ideas in nerd communities while being technically incompetent has been obsessed with for nearly thirty years. The reason I forced myself to think on it enough to feel okay arguing against it was due to being successfully convinced about EA’s value when I was a child,[2] keeping up with it in adolescence, and noticing that x-risk-mitigation advocates successfully started diverting a bunch of funding from malaria nets to “extinction prevention,” by making arguments around EV, which in my mind (after much consideration) is likely going to go down as one of the biggest unforced mistakes in human history. This is neither here nor there, though;[3] just relevant background.
This arms race framing is, transparently, used by major labs as a way to justify and provoke their intense valuations and immense spending on attempting to build the thing that you’re worried about. It’s a way of red-teaming venture and human capital. “The first person to build it controls the entire future, so we need to have a good actor doing it. Give us ten trillion dollars.”[4] The only person to have participated in the founding of nearly every major lab (Musk—OpenAI->Anthropic, xAI) publicly acknowledged the fact that he met his ex-partner off a discussion of Roko’s Basilisk. More people are familiar with the argument in its primary form of “justifying bad actions from bad actors” than its secondary form.
I’m not making an argument that you should stop talking about extinction vis-à-vis lawmakers. I don’t think it’s as persuasive as you’re making the case that it is, and I think that there are many things lawmakers largely agree on but don’t have the political will from the wider public to execute on, but I don’t fault trying to convince lawmakers as a strategy in itself. However, the graph you linked earlier just shows awareness of the argument, it doesn’t show alignment or agreement with it, nor whether any of the lawmakers who have made those arguments would actually pass policy proposed to mitigate x-risk. It also (I believe) is likely going to look a lot worse after midterms come around.
There are five hundred and thirty-five members of Congress; thirty are talking about, not necessarily aligned with, the concept of AGI risk. Even assuming the current trend scales linearly, which doesn’t seem likely to me, it still doesn’t seem to be enough with the timelines the people who made the graph are projecting. Appealing to populism to pass stalling legislation, given how far ahead negative views of AI among the public are relative to lawmakers’ views on artificial intelligence, even if your goal is still ultimately a complete ban, would if nothing else likely buy you the time to convince lawmakers of what you aim to convince them of. Buying yourself another 5.5 months, under the implication of the graph, would literally double congressional awareness, being dramatically more effective (x2 of the end result of your efforts without it) than just trying to convince lawmakers, and reducing the effectiveness of LLM companies using populist policy and regulation would likely offer significantly more time than an extra 5.5 months. Popping the bubble would take a lot of people’s compensation with it, pushing the economy to reallocate resources to more productive uses of time than literally killing everyone.
My claim is more that efforts to convince the wider public, not lawmakers, has been mislaid, and that the energy would be better shifted in a marginally different direction. You’re unlikely to successfully convince the wider public; they’ve been continuously inoculated against what you want them to believe, by the ever-persistent, ever-aware-of-x-risk voices of the people currently causing the problems at hand. If the people making an argument are people you absolutely despise, who you can see speak from both sides of their mouth,[5] it’s far less likely to gain any ground in your mind. This has been the public’s current exposure to x-risk arguments: They exist to increase animosity to generative AI, due to the sheer hypocrisy of the players involved making them, not to gain purchase due to merit but rather blunt force of capital.
It’s also worth acknowledging that this post was, admittedly, a very silly way of talking about the art of convincing people in public in general, quite literally inspired by a street I walk on every day being filled with literal petitioners, who are trying to achieve much easier things, who still often fail at what they’re trying to do, because they don’t understand how persuasion works. It applies to lawmakers, as well, but lawmakers are closer to you than you are to the public. The “Save BART” people are actually a good example of inefficiently trying to compel the public into action (though I wouldn’t be surprised if they succeed anyway); it is nearly impossible to convince people to sign a petition off of love for something, but it is extremely easy to convince someone to do something out of spite. Fear is somewhere in the middle; accepting that you’re afraid is in a sense admitting to cowardice. Most people are averse to this, unless it’s really bad, so it’s very hard to convert fear into action. Easier than to convert love into action, though.
The strategies to use in private are different than the ones to use in public, so the post doesn’t bother to target that use-case. I would likely write a different post if writing advice specifically for your scenario of convincing lawmakers, but I’ve never been around lawmakers to verify my suspicions. It does sound like a fun game, though; at some point I should figure out something fun to try and lobby for, do so, and then write that post after I’ve proven my point. It seems likely that you’re not doing something correctly, if there are only thirty American Congresscritters who have bought in (and seeing how success leans much heavier toward the American right-wing in recent memory, per the graph), but I can’t really diagnose it without either seeing what people are trying or trying to do it myself and seeing what the failure cases are. I have some existing priors about how Congressional persuasion happens, based on what I’ve read from and of people who have successfully done it (not in the x-risk space), but I have no way of validating my suspicions around it at this time.
The post never actually encourages lying, it merely encourages not leading with the high-friction parts. Convincing people, especially in public, is primarily about reducing the friction to agreement.[6] Early existential risk arguments caught on largely by this method, by making it high status to signal agreement with it. In the awkward middle period we’re in, making it profitable for high-value targets is the current meta. However, status games don’t work as well on the public at large, and there’s not enough money to go around to incentivize everyone in the way the movement is currently seducing high-value targets with.
Having moved to SF recently, I’ve been made pretty viscerally aware of how rationalism has sort of failed in a place that should hold its apex. Everyone speaks with a rationalist tongue, but most of them only stole the rhetoric from it. People who should be halfway to your beliefs are instead of the belief that AI will somehow, or is, making everything better; describing silly fantasy scenarios of benevolent futures while consciously rejecting your view and mistakenly thinking mine is novel. It’s pretty puzzling to observe firsthand.
I think it’s primarily due to the public acknowledging that there’s something compelling about rationalism based off of interaction with rationalists, but failing to actually grab onto any of the thinking associated with it, largely due to poor public communication on behalf of the rationalists rather than active rejection. This, plus motivated reasoning with an assumption that the efficient market hypothesis is correct, has made their thinking even sloppier than it would have been without the exposure to rationalism. This seems bad to me. To convince people of a point, you don’t need an in-person conversation. In-person conversations make it a lot easier, but people successfully convince the public of things all the time. The failure of public communication on this axis feels a bit like a critical mistake, and one that was probably avoidable with different messaging.
Convincing guests on the JRE was fine, but the fact that they’ve been talking about it for years now on what is one of the biggest soap boxes in the world and have convinced exceedingly few regular people is a sign that the public messaging just isn’t working, and manufacturing public support is important to passing legislation.
Rationalists should win, and while I’m not a rationalist and I don’t care too much about winning,[1:1] I do think the strategy outlined in the post offers a much better approach to public messaging for rats, if they actually want to win in the long run on this issue, than the present approach, which is actively self-defeating.
circa nine years old around 2013 or so, which I’ll admit is probably out of distribution for exposure to x-risk arguments; I am not necessarily representative of the normal mode of discovering x-risk arguments. I consider myself to be fairly normal, though, as an individual.
I will never forgive x-risk mitigation advocates for reducing my ability to use em dashes in public writings by virtue of causing the companies responsible for the creation of LLMs to get funding. I miss using em dashes. This is objectively worse than human extinction would be. I wanted to use one here, but, well, you know. I guess I could start misusing spaced en dashes, but it isn’t the same.
Does this seem familiar? Which lab head am I referencing? Just kidding; it’s all of them, from Altman to Legg[7] to Amodei to Musk. They all make exactly this argument in the same hushed and worried tone while excitedly ushering in the same future they’re allegedly afraid of.
A quote that comes to mind with how these people have historically advocated for their position is Machiavelli paraphrasing the Romans: “As for that which has been said, that it is better and more advantageous for your state not to interfere in our war, nothing can be more erroneous; because by not interfering you will be left, without favour or consideration, the guerdon of the conqueror.”
In private, or semi-private contexts, strangely, it’s often the opposite. It’s very useful to start with something outlandish, the least-charitable-to-yourself way of presenting your views while still remaining honest (to quote Yudkowsky paraphrasing LeGuin, “Don’t say things that are literally false.”), that you’ll never in a million years convince someone of, and make a game out of convincing your target the halfway point is reasonable that way. This doesn’t really work in public, but I think it’s why Yudkowsky leans so heavily on it in public: It works really well in a semi-private context like 2014 lesswrong, or in the comments of a −14 2026 lesswrong post.[8] There’s probably a decent post to be written solely about this weird quirk of persuasion, maybe titled something like “Boiling the Frog” or something of that nature, but I think mechanically, if not getting into the “why” of how it works, it’s simple enough to be described by taking a relatively popular internet macro quote describing a bad faith action, and just generalizing it a bit beyond bad faith.
I am, of course, referring to the Moxon quote that may have came to mind:
Meet me in the middle, says the unjust man.
You take a step towards him, he takes a step back.
Meet me in the middle, says the unjust man.
This doesn’t have to be an intentionally bad-faith action, and doesn’t actually require the act of stepping back to still be successful. Zeno’s dichotomy paradox points out that there are an infinite number of half-steps to get to any point. By repeatedly arguing your point, but staying stable in it, you draw your target ever-closer, even as you stay still and continuously take good faith effort.
I think there are two types of natural reactions upon first exposure to lesswrong/Bayesian rationalism among people who eventually find themselves convinced of what’s within; there are people who find the tenets of it obvious, and there are people who find themselves with an initial aversion to it in a way that reflects the sort of mindset you’d have in a debate club, that dissolves upon further exposure and recursive argumentation. This latter mechanism, I think, though foreign to me (when I was exposed to rationalism I was very much in the former camp), is ultimately the same one as what I’m describing.
It also happens to be really fun, which is motivating in itself, beyond anything about efficacy, and perhaps in conflict with it: I know that I, for example, will often argue in public in a way that’s primarily useful in private, just because it’s a lot more fun, to the detriment of the point I’m making.
Given that the recommendations here are explicitly about convincing members of the public in a public setting, and I am currently aiming to convince policy people in a private setting, I think the advice here isn’t as applicable to what I, personally, am trying to do in the next few months. Thanks for everything you’ve written, and I’ll keep it in mind if that changes. I am going to bow out of the thread at this point, but didn’t want to just disappear, given the effort you clearly put in. Thanks for the discussion—I quite appreciate it!
I appreciate the effort you’re putting into this thread here. While I haven’t changed my mind on high-level strategy, I have learned at least a couple of things here, so I do want to reassure you that your efforts aren’t just hitting a brick wall here.
That said, let’s take a look at one of your quotes: “Human extinction isn’t that far out of the Overton Window at this point; people like Musk creating big headlines talking about it every year has moved it firmly adjacent to it, if not inside. It’s half of what gets talked about nowadays on some of the most-listened-to platforms in the world, like the Joe Rogan Experience, whose audience is most certainly not an educated one.”
The main question is—how do you think this happened? Human extinction is weird and out there. You yourself don’t believe it. But you know about it, which at least gives you the possibility of agreeing or disagreeing with us. This happened through awareness building. I don’t think this happened as an inevitable result of AI’s ascendancy. I think LessWrong style advocacy around extinction risk was a direct contributor, and the ability to say something like “X-risk is half of what gets talked about nowadays in the Joe Rogan Experience” is what it looks like for our advocacy to be succeeding.
Furthermore, I don’t think we’ve hit steeply diminishing returns to awareness. This post https://www.lesswrong.com/posts/A7BtBD9BAfK2kKSEr/what-we-learned-from-briefing-140-lawmakers-on-the-threat is pretty enlightening here—it’s not too out of date (Sep 2024 - Feb 2026) and describes that most British MP’s are unaware of basic facts that support our argument. I think a lot of people who disagree with our argument do so not because they’ve developed detailed models of how we view the situation and they think we’re wrong on certain key points. I think they disagree because a sensible prior is “This technology will not kill everyone” and they haven’t heard enough information to make them consciously reconsider this. There is actually key information that is required for the extinction argument to make sense that many people don’t know. I don’t think our arguments have failed to connect with all such people. I think they’ve often simply never been made in person with sufficient fidelity to drive our point home.
Note that I am not accusing you personally of this. It is entirely possible to understand our argument well and still disagree with it, and some such people exist, and may be basically unreachable as you say. But I think this does not yet account for most of the important people who disagree with us.
There’s also common knowledge—it takes an unusually courageous person to stand up first. Asking someone “Please stand up against extinction risk, even though you may be alone and the confused silence will be deafening” is a much harder ask than “Please join our large and growing group of people who are openly concerned about extinction risk and lend your voice to the crowd” and we appear to be moving rather rapidly towards the latter. Anyone who is convinced by the second but not the first is very far from unreachable.
I think your strategy has non-zero value, in the sense that your strategy is better than doing nothing. But I don’t think it’s better than what we’re doing. I think that:
We have successfully changed extinction risk from AI from “A thing nobody has heard about” to “A thing that is discussed in halls of power and major media”, and this is a big change.
There is still a lot of utility left in educating people—reaching new people and making the case to people who have vaguely heard about extinction risk. Most people who disagree with us don’t deeply disagree with us to the point of being unreachable.
The best way to educate people about extinction risk is to talk directly about extinction risk.
This has higher value than attempting to use existing prosaic anti-AI sentiment to push obstructionist regulation. Although I have been convinced this is a good (i.e +EV) strategy, I don’t think it is the best strategy.
Thanks for the acknowledgement that I’m not hitting a brick wall! I’d happily die on any hill, [1] but it’s always more enjoyable when dying on it to open ears.
The reason I was exposed to arguments around x-risk is the same reason that I’d imagine most people who have been actively making the “problem” worse know about it: It’s something that a fanfiction author who was unusually effective at popularizing ideas in nerd communities while being technically incompetent has been obsessed with for nearly thirty years. The reason I forced myself to think on it enough to feel okay arguing against it was due to being successfully convinced about EA’s value when I was a child, [2] keeping up with it in adolescence, and noticing that x-risk-mitigation advocates successfully started diverting a bunch of funding from malaria nets to “extinction prevention,” by making arguments around EV, which in my mind (after much consideration) is likely going to go down as one of the biggest unforced mistakes in human history. This is neither here nor there, though; [3] just relevant background.
This arms race framing is, transparently, used by major labs as a way to justify and provoke their intense valuations and immense spending on attempting to build the thing that you’re worried about. It’s a way of red-teaming venture and human capital. “The first person to build it controls the entire future, so we need to have a good actor doing it. Give us ten trillion dollars.” [4] The only person to have participated in the founding of nearly every major lab (Musk—OpenAI->Anthropic, xAI) publicly acknowledged the fact that he met his ex-partner off a discussion of Roko’s Basilisk. More people are familiar with the argument in its primary form of “justifying bad actions from bad actors” than its secondary form.
I’m not making an argument that you should stop talking about extinction vis-à-vis lawmakers. I don’t think it’s as persuasive as you’re making the case that it is, and I think that there are many things lawmakers largely agree on but don’t have the political will from the wider public to execute on, but I don’t fault trying to convince lawmakers as a strategy in itself. However, the graph you linked earlier just shows awareness of the argument, it doesn’t show alignment or agreement with it, nor whether any of the lawmakers who have made those arguments would actually pass policy proposed to mitigate x-risk. It also (I believe) is likely going to look a lot worse after midterms come around.
There are five hundred and thirty-five members of Congress; thirty are talking about, not necessarily aligned with, the concept of AGI risk. Even assuming the current trend scales linearly, which doesn’t seem likely to me, it still doesn’t seem to be enough with the timelines the people who made the graph are projecting. Appealing to populism to pass stalling legislation, given how far ahead negative views of AI among the public are relative to lawmakers’ views on artificial intelligence, even if your goal is still ultimately a complete ban, would if nothing else likely buy you the time to convince lawmakers of what you aim to convince them of. Buying yourself another 5.5 months, under the implication of the graph, would literally double congressional awareness, being dramatically more effective (x2 of the end result of your efforts without it) than just trying to convince lawmakers, and reducing the effectiveness of LLM companies using populist policy and regulation would likely offer significantly more time than an extra 5.5 months. Popping the bubble would take a lot of people’s compensation with it, pushing the economy to reallocate resources to more productive uses of time than literally killing everyone.
My claim is more that efforts to convince the wider public, not lawmakers, has been mislaid, and that the energy would be better shifted in a marginally different direction. You’re unlikely to successfully convince the wider public; they’ve been continuously inoculated against what you want them to believe, by the ever-persistent, ever-aware-of-x-risk voices of the people currently causing the problems at hand. If the people making an argument are people you absolutely despise, who you can see speak from both sides of their mouth, [5] it’s far less likely to gain any ground in your mind. This has been the public’s current exposure to x-risk arguments: They exist to increase animosity to generative AI, due to the sheer hypocrisy of the players involved making them, not to gain purchase due to merit but rather blunt force of capital.
It’s also worth acknowledging that this post was, admittedly, a very silly way of talking about the art of convincing people in public in general, quite literally inspired by a street I walk on every day being filled with literal petitioners, who are trying to achieve much easier things, who still often fail at what they’re trying to do, because they don’t understand how persuasion works. It applies to lawmakers, as well, but lawmakers are closer to you than you are to the public. The “Save BART” people are actually a good example of inefficiently trying to compel the public into action (though I wouldn’t be surprised if they succeed anyway); it is nearly impossible to convince people to sign a petition off of love for something, but it is extremely easy to convince someone to do something out of spite. Fear is somewhere in the middle; accepting that you’re afraid is in a sense admitting to cowardice. Most people are averse to this, unless it’s really bad, so it’s very hard to convert fear into action. Easier than to convert love into action, though.
The strategies to use in private are different than the ones to use in public, so the post doesn’t bother to target that use-case. I would likely write a different post if writing advice specifically for your scenario of convincing lawmakers, but I’ve never been around lawmakers to verify my suspicions. It does sound like a fun game, though; at some point I should figure out something fun to try and lobby for, do so, and then write that post after I’ve proven my point. It seems likely that you’re not doing something correctly, if there are only thirty American Congresscritters who have bought in (and seeing how success leans much heavier toward the American right-wing in recent memory, per the graph), but I can’t really diagnose it without either seeing what people are trying or trying to do it myself and seeing what the failure cases are. I have some existing priors about how Congressional persuasion happens, based on what I’ve read from and of people who have successfully done it (not in the x-risk space), but I have no way of validating my suspicions around it at this time.
The post never actually encourages lying, it merely encourages not leading with the high-friction parts. Convincing people, especially in public, is primarily about reducing the friction to agreement. [6] Early existential risk arguments caught on largely by this method, by making it high status to signal agreement with it. In the awkward middle period we’re in, making it profitable for high-value targets is the current meta. However, status games don’t work as well on the public at large, and there’s not enough money to go around to incentivize everyone in the way the movement is currently seducing high-value targets with.
Having moved to SF recently, I’ve been made pretty viscerally aware of how rationalism has sort of failed in a place that should hold its apex. Everyone speaks with a rationalist tongue, but most of them only stole the rhetoric from it. People who should be halfway to your beliefs are instead of the belief that AI will somehow, or is, making everything better; describing silly fantasy scenarios of benevolent futures while consciously rejecting your view and mistakenly thinking mine is novel. It’s pretty puzzling to observe firsthand.
I think it’s primarily due to the public acknowledging that there’s something compelling about rationalism based off of interaction with rationalists, but failing to actually grab onto any of the thinking associated with it, largely due to poor public communication on behalf of the rationalists rather than active rejection. This, plus motivated reasoning with an assumption that the efficient market hypothesis is correct, has made their thinking even sloppier than it would have been without the exposure to rationalism. This seems bad to me. To convince people of a point, you don’t need an in-person conversation. In-person conversations make it a lot easier, but people successfully convince the public of things all the time. The failure of public communication on this axis feels a bit like a critical mistake, and one that was probably avoidable with different messaging.
Convincing guests on the JRE was fine, but the fact that they’ve been talking about it for years now on what is one of the biggest soap boxes in the world and have convinced exceedingly few regular people is a sign that the public messaging just isn’t working, and manufacturing public support is important to passing legislation.
Rationalists should win, and while I’m not a rationalist and I don’t care too much about winning, [1:1] I do think the strategy outlined in the post offers a much better approach to public messaging for rats, if they actually want to win in the long run on this issue, than the present approach, which is actively self-defeating.
To quote a piece of art I like, “I think I’m right; I don’t think it matters.”
circa nine years old around 2013 or so, which I’ll admit is probably out of distribution for exposure to x-risk arguments; I am not necessarily representative of the normal mode of discovering x-risk arguments. I consider myself to be fairly normal, though, as an individual.
I will never forgive x-risk mitigation advocates for reducing my ability to use em dashes in public writings by virtue of causing the companies responsible for the creation of LLMs to get funding. I miss using em dashes. This is objectively worse than human extinction would be. I wanted to use one here, but, well, you know. I guess I could start misusing spaced en dashes, but it isn’t the same.
Does this seem familiar? Which lab head am I referencing? Just kidding; it’s all of them, from Altman to Legg [7] to Amodei to Musk. They all make exactly this argument in the same hushed and worried tone while excitedly ushering in the same future they’re allegedly afraid of.
A quote that comes to mind with how these people have historically advocated for their position is Machiavelli paraphrasing the Romans: “As for that which has been said, that it is better and more advantageous for your state not to interfere in our war, nothing can be more erroneous; because by not interfering you will be left, without favour or consideration, the guerdon of the conqueror.”
In private, or semi-private contexts, strangely, it’s often the opposite. It’s very useful to start with something outlandish, the least-charitable-to-yourself way of presenting your views while still remaining honest (to quote Yudkowsky paraphrasing LeGuin, “Don’t say things that are literally false.”), that you’ll never in a million years convince someone of, and make a game out of convincing your target the halfway point is reasonable that way. This doesn’t really work in public, but I think it’s why Yudkowsky leans so heavily on it in public: It works really well in a semi-private context like 2014 lesswrong, or in the comments of a −14 2026 lesswrong post. [8] There’s probably a decent post to be written solely about this weird quirk of persuasion, maybe titled something like “Boiling the Frog” or something of that nature, but I think mechanically, if not getting into the “why” of how it works, it’s simple enough to be described by taking a relatively popular internet macro quote describing a bad faith action, and just generalizing it a bit beyond bad faith.
I am, of course, referring to the Moxon quote that may have came to mind:
This doesn’t have to be an intentionally bad-faith action, and doesn’t actually require the act of stepping back to still be successful. Zeno’s dichotomy paradox points out that there are an infinite number of half-steps to get to any point. By repeatedly arguing your point, but staying stable in it, you draw your target ever-closer, even as you stay still and continuously take good faith effort.
I think there are two types of natural reactions upon first exposure to lesswrong/Bayesian rationalism among people who eventually find themselves convinced of what’s within; there are people who find the tenets of it obvious, and there are people who find themselves with an initial aversion to it in a way that reflects the sort of mindset you’d have in a debate club, that dissolves upon further exposure and recursive argumentation. This latter mechanism, I think, though foreign to me (when I was exposed to rationalism I was very much in the former camp), is ultimately the same one as what I’m describing.
Who, coincidentally, has been projecting 2028 as his 50% confidence target for AGI and has been on the existential risk train since at least 2011.
It also happens to be really fun, which is motivating in itself, beyond anything about efficacy, and perhaps in conflict with it: I know that I, for example, will often argue in public in a way that’s primarily useful in private, just because it’s a lot more fun, to the detriment of the point I’m making.
Given that the recommendations here are explicitly about convincing members of the public in a public setting, and I am currently aiming to convince policy people in a private setting, I think the advice here isn’t as applicable to what I, personally, am trying to do in the next few months. Thanks for everything you’ve written, and I’ll keep it in mind if that changes. I am going to bow out of the thread at this point, but didn’t want to just disappear, given the effort you clearly put in. Thanks for the discussion—I quite appreciate it!