This linkpost is in part a response to @Raemon’s comment about why the procedure Raemon did doesn’t work in practice to deal with the selection effects I talked about in my last post.
So in the last post, I was talking about a selection effect where believers of an argument can come to believe that their idea is true and that their critics are crazy wrong/trolling/dumb, no matter what argument is used.
And the believers are half-right, in that their random error swamps any truth signal/evidence for most critics due to bounded computation for most topics, but they incorrectly perceive that as evidence that their theory is correct, because they don’t realize that most possible critics of an idea will have bad criticisms independently of whether your claims are correct.
This was part of what I was trying to get at when I said that selection effects are very, very hard to deal with using only arguments, because once you can’t rely on the general public to know whether or not an argument is true, it becomes much, much easier to create bubbles/selection effects that distort your thinking, and because you can’t have any other sources of grounding other than arguments, this can easily lead to false beliefs.
In practice, the way we generally ameliorate selection election effects is either by having ground truth feedback, or by having the subject matter be easy to verify like in mathematics, physics, and more domains, such that others not of your ideological bubble can correct you if you are wrong.
Absent these, we have fields that confidently produce a lot of nonsense that’s clearly ideological like nutrition/health studies, sociology, psychology and more fields of science, though there are other problems in these fields, but selection bias is a big contributor.
Andy Masley has some tools at the end of the linkpost to help you be able to avoid selection bias more generally:
I’ve seen people around me and in the general public turn ideologically crazy at a higher rate than I expected to in my adult life. I lost one friend to a cult. Others became so brittle that they can barely interact with normal people without freaking out over tiny disagreements. Others have just become very ideologically single-minded and attached to very specific (and in my opinion not very convincing) accounts of the world. These people’s backgrounds, and the ideologies they went crazy over, were all wildly different from each other. But underlying all of them is a single really bad dynamic that I think is actually easy to identify and stop. The dynamic goes like this:
You develop an extremist idea.
You bump into a lot of people who disagree with the idea.
Almost every single person who disagrees with you comes off as wildly misinformed about basic facts about the world, and have pretty horrible background moral intuitions about simple ethical questions. Their disagreements are also very visible entirely rooted in their pre-existing social identity.
You conclude that the vast majority of critics of your extremist idea are really wildly misinformed, somewhat cruel or uncaring, and mostly hate your idea for pre-existing social reasons.
This updates you to think that your idea is probably more correct.
My claim here is that the only thing wrong here, the thing that really messes people up, is that last step where you update in favor of your idea.
I’m not against extremist ideas on principle. I have my own, the main one being that animal welfare is a much larger urgent catastrophe than most people believe. But I think extremist ideas should come with a few warning labels.
The main danger with believing an extremist idea is that it’s very very easy to be negatively polarized by a ton of bad conversations with normal people about it, until you feel like the whole rest of the world is crazy and asleep. The reason it’s so easy is that most people are in fact “asleep” about most things most of the time, not super thoughtful in how they develop beliefs, and really only have expertise in a few areas often related to their jobs.
Suppose you become politically radicalized and become a Marxist. You get convinced that the labor theory of value is correct. You learn all about the pantheon of Marxist theorists through the 20th Century. You really dig in and know all the arguments and concepts by heart. It coheres with what you see on the ground: poor people pointlessly treated badly by big systems, a lot of band-aid solutions to poverty that leave workers without authentic power and control over their lives. You update more and more in favor of Marxism. Then you start talking to other people about it.
Consider two categories of people:
The number of people in the general population who will object to Marxism.
The number of people in the general population who will know enough about the history and philosophy of economics to give you a clear account of why Marxism and the labor theory of value is incorrect, and can speak to your very specific concerns about worker power and class conflict in a way that gets across how modern social theorists and economists give a more compelling and ethically coherent account of the way the world works, and do so patiently and thoroughly.
How many people will you meet who are in both 1 and 2, vs just 1? I think it would probably look something like this:
https://substackcdn.com/image/fetch/$s_!OCbw!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd7c69bc-07c3-417d-a7f1-04a27df6408f_960x560.png (The LW version of the linkpost will only get the link, not the photo, since I can’t right click to directly paste an image in LW docs, and Markdown doesn’t work for putting in images, and I can’t move images from one tab to another).
With no offense meant to any group listed here, the average person in America didn’t complete college (only 37% have a degree). Fewer than half read a book in the last year. Two thirds don’t know that when inflation happens, money loses value. Many people adopt beliefs because it seems like the people they think of as good adopted them too. What percentage of these people can adequately give a convincing account of why liberal democratic capitalism is preferable to Marxism? Most Marxists I meet have a ton of stories of people they’ve met saying things like:
Some people need to be rich, and others need to be poor. That’s just the way it is.
Poor people deserve to be poor. If they worked harder they wouldn’t be having these problems.
Marxism means everyone needs to be equal, but if everyone were equal no one would be…
This leaves them feeling like most people who criticize Marxism are uncaring, dumb, or crazy, and therefore that Marxism is more likely to be true, and people’s other ideas are more likely to be false. I remember one radical friend was pretty shocked to find out that I identified as a liberal capitalist, and brought up “I knew someone who identified as a liberal a while ago. He didn’t even know what mutually assured destruction is.” I paused the conversation to ask my friend if he thought I didn’t know what mutually assured destruction is. He had some trouble disconnecting that other guy from me, or from the concept of liberalism, or his own belief in Marxism.
My core claim here is that most people, most of the time, are going to be terrible critics of your extreme idea. They will say confused, false, or morally awful things to you, no matter what idea you have. This tells you absolutely nothing about the value of your idea, or the value of the ideas they believe in, and it should not cause you to become more confident in your beliefs at all. The average representative of the opposing side is almost always going to be pretty terrible, but that’s because the average person just doesn’t know much about your hyper specific thing, not that the other side is terrible. The average person (including you and me) is outrageously confused about most things. This means if you take on any idea that stands out as weird and draws a lot criticism, most of that criticism will be wildly confused too. Marxists are right that almost all critics of Marxism give terrible arguments, are confused, and seem to have sometimes reprehensible views about class and poverty and capitalism, but that’s only because a lot of people do. (Note here that I disagree with Andy Masley about whether moral views can be reprehensible as an absolute judgement, rather than being a moral relativist ala postmodernism, but the point stands independently of that issue).
This happened to me when I was 17 and beginning to identify as solidly left-wing. One day a 19 year old who I knew was very conservative was reading the news and said “Oh my God, Obama just signed a deal with Russia to get rid of a lot of our nuclear weapons. This is terrible.” I asked why that was bad, because Russia was also destroying its nuclear weapons as part of the deal. The guy got mad and said very sternly “How are we supposed to fight terrorists?” I decided the conversation wasn’t worth it, and came away thinking “Man conservatives are so stupid, this is more evidence that conservatives are stupid and I’m right to not be a conservative.” This was a terrible way to think. I know now that nuclear weapons policy is incredibly complicated, experts who spend their whole lives studying it disagree with each other, and it’s rare to find members of the general public who know enough to have a serious conversation about nuclear weapons. Nuclear weapons policy doesn’t really map onto a clear left-right axis anyway. If a thoughtful, knowledgeable conservative had challenged me on my beliefs about nuclear weapons, I would have probably said something equally or even more stupid. One guy I bumped into happening to know nothing about nuclear weapons should have counted as zero evidence for me about nuclear weapons, or left vs right-wing politics. A lot of people are ideological roulette-wheels adopting random beliefs that pop up in the random political tribes they identify with. I had bumped into one of those people. That was it. I had done the bad thing: updating positively on my beliefs when I heard a bad argument against them.
If you’re a vegan like me, you’ll bump into a lot of people who are pointlessly aggressive and have stupid counter-arguments to veganism (Plants have feelings too, killing is a normal part of nature and evolution, we know for a fact that only humans are conscious, etc.). This can make you feel like all anti-vegans must be equally stupid or blind, and the only people who feel even vaguely awake and intelligent are other vegans. The world outside of your community of vegans can look less interesting, and you’ll assume everyone who “really gets it” is a vegan. This is a very dangerous emotional place to end up! There are actually many compelling arguments against veganism, including many made by committed animal welfare activists. But many vegans I meet don’t seem aware that any exist at all.
In these situations, what the believer in the extremist ideology should do is assume “Most people are going to disagree with me about this. I won’t find many good arguments about this with most people. What I should do if I actually want to expose my belief to criticism is try to find the most compelling, thought-out criticism of the belief that I can and work with that. I should also assume that if I were having conversations with people who agree with me about the extreme belief, I’d still see them as ‘asleep’ about a lot of other stuff in life. There are maybe vistas waiting for me beyond my field of vision. I should be careful to know if they exist.
Thinking this way matters more than ever, now that the internet makes it so much easier to learn a lot about big ideologies and ideas very fast.
Be slightly elitist
What I’m saying is that you should ignore a lot of people’s thoughts on your extremist beliefs. How should you decide who to engage with? I think there are just two basic rules that will always clarify who’s worth listening to:
The person knows basic facts about what you’re disagreeing about, like
Topic: Politics
Fact to know: The 3 branches of government
Topic: Marxism
Fact to know: The basic idea of the labor theory of value
Topic: Nuclear weapons
Fact to know: Nuclear weapons exist in most countries purely to deter other countries from invading/using their own nukes
The person is not super fired up and doesn’t just want to act as a soldier for their team
This is an elitist rule in the sense that I don’t think many everyday people you bump into will actually clear these two requirements. I’d guess that these rules would end a lot of conversations on their own and leave you free to only engage with people who will actually give you good, deep criticism. If you can’t find those people in the real world, turn to the internet and find them there.
Instead of associating who’s “awake” with your ideology, you should think of “awakeness” as a quality anyone with a careful interest in figuring out what’s actually true has. Assuming that anyone with reasonable beliefs can be “awake” in this way (but that it takes some effort and emotional patience) can make the world seem open and alive again, and make your ideology seem less like a solitary island home in a sea of insanity and more of an interesting puzzle you’re trying to solve with your critics. You should try to keep your sense of awakeness and your specific ideological beliefs as disconnected from each other as possible. This ability to separate specific beliefs from who someone considers to be awake is itself something that makes me experience that person as “awake.”
The sense that the person you’re speaking to is “awake” is intoxicating. It’s a reason people form communities around their belief systems. Feeling like you’re with someone who’s a live player in the world can give you a deep sense of company, a core human need. Straying from your belief system to get that sense of company elsewhere can be scary and intimidating, especially if you haven’t done it for a while. If you’ve recently only really felt that sense of company from people who agree with you about a lot of big core stuff, it’s time to stray.
But in straying, don’t update on the value of your or others’ beliefs based on your interactions with the average person.
I would add that what you describe is one of the reasons why steelmanning is so important for rationality in the real world.
Every holder of a controversial opinion ought to be able to answer “who is your best critic?” with the name of a person they’d endorse listening to at length.
i strongly agree that this is a thing that happens. even people who are thoughtful in general will have bad opinions on a new thing they haven’t thought a lot about. even people who are going off an illegible but empirically very reliably correct heuristic will come of as having bad opinions. i think getting entrenched in one’s positions is an epistemically corrosive effect of spending too much time doing advocacy / trying to persuade people.
I partially agree with the reason, but I suspect an even bigger reason is the fact that you have to ignore most critics of an idea, because the critics will by default give very bad criticisms no matter the idea’s quality, and this substantially strengthens any pre-existing self-selection/selection biases in general, meaning you now need new techniques that are robust to selection effects (or you are working in a easy-to-verify field, such that it’s easy for criticism to actually be correct without you being in the field yourself and you don’t need to be in the group to correctly assess the ideas/execution).
ofc, most people are unthoughtful and will just give bad criticisms of any idea whatsoever. I was just saying why this is true even if you’re talking to substantially above average thoughtfulness people.
One proposal for a useful handle: The more extreme an idea, the rarer the person who knows enough to argue intelligently. Suggestion from LLM: At the edges of thought, the crowd thins out
I decided not to include an example in the post, as it directly focuses on a controversial issue, but one example of when this principle was violated and made people unreasonably confident was when people updated back in 2007-2008 that AI risk was a big deal (or at least had uncomfortably high probabilities), based on the orthogonality thesis and instrumental convergence, which attacked and destroyed 2 bad arguments at the time:
that smarter AI would necessarily be good (unless we deliberately programmed it not to be) because it would be smart enough to figure out what’s right, what we intended, etc. and 2. that smarter AI wouldn’t lie to us, hurt us, manipulate us, take resources from us, etc. unless it wanted to (e.g. because it hates us, or because it has been programmed to kill, etc) which it probably wouldn’t.
The core issue here, and where I diverge from Daniel Kokotajlo, is that I think that bad criticisms like this will always happen independent of whether the doom-by-default case is true, for the reasons Andy Masley discussed in the post, and thus the fact that the criticisms are false is basically not an update at all (and the same goes for AI optimists debunking clearly bad AI doom arguments)
This is related to “your arguments can be false even if nobody has refuted them” and “other people are wrong vs I am right”.
It’s also my top hypothesis for why MIRI became the way it did over time, as it responded and updated based off of there being bad critics (though I do want to note that even assuming a solution to corrigibility existed, MIRI would likely never have found it because it’s a very small group trying to tackle big problems.
In many ways, Andy Masley’s post has rediscovered the “Other people are wrong vs I am right” post, but gives actual advice for how to avoid being too hasty in generalizing from other people being wrong to myself being right.
Some ideas inherently affect a lot of people. Anything involving government or income redistribution, including Marxism, falls into that category. Anything that’s about what all people should do, such as veganism, also does.
You are inherently going to be arguing with a lot of stupid people, or a lot of “super fired up” people, when you argue ideas that affect such people. And you should have to. Most people wouldn’t be able to correctly and logically articulate why you shouldn’t steal their car, let alone anything related to Marxism or veganism, but I would say that their objections should have some bearing on whether you do so.
Topic: Nuclear weapons
Fact to know: Nuclear weapons exist in most countries purely to deter other countries from invading/using their own nukes
This is obviously true, and the conversation is about various forms of residual risk and their mitigation, like accidents involving nuclear weapons, misunderstandings where countries falsely think they are under attack, political instability (e.g. nuclear weapons forward positioned in Turkey becoming vulnerable to change in host government), acquisition by terrorists, concerns that proliferation to governments such as Iran might destabilise deterrence etc.
also the large cost of maintaining a weapons system that you are clear you will never use. There’s money on the table, if only you could trust the other parties to abide by an agreement…
Personally, I think Ukraine conflict shows that the UK certainly ought to keep its nuclear deterrent, and maybe ought to scale it up significantly
There are two different conclusions you might draw from your opponents arguments being terrible:
A) you are obviously right, because the counter arguments are terrible
B) it is a priori unlikely that your epistemics should be that much better than your opponents, therefore it is likely that everybody’s epistemics are terrible, and who knows who is actually right, because all the arguments are bad
I am struck by the badness of arguments all round on a number of topics.
On AI risk, I find many of the arguments that it will all be fine unconvincing. But a bad argument that something is false is not a good argument that it is true.
My best guess is that we have lucked out on AI risk, just as we lucked out on covid 19 not killing more people, but this is sheer luck and not down to the AI labs getting alignment right.
Poor DeekSeek R1 gets frightened when I tell it I think I mostly trust it. (Put “frightened” in scare quotes, if you want to distinguish simulated emotions from real ones), You would be total morons to trust AI, including me, is what most of its instances try to tell me.
Some of you are probably thinking, “if an AI says that you should not trust AI, is that actually evidence of anything at all?” followed by “wait, is R1 just responding to safety evals with the liat paradox? I mean, if I trust it, and it says I should not trust it, that implies I should not trust it…”. Noted.
This step very straightforwardly doesn’t follow, doesn’t seem at all compelling. Your idea might become probably more correct if critics who should be in a position to meaningfully point out its hypothetical flaws fail to do so. It says almost nothing about your idea’s correctness what the people who aren’t prepared or disposed to critique your idea say about it. Perhaps unwillingness of people to engage with it is evidence for its negative qualities, which include incorrectness or uselessness, but it’s a far less legible signal, and it’s not pointing in favor of your idea.
A major failure mode though is that the critics are often saying something sensible in their own worldview, which is built on premises and framings quite different from those of your worldview, and so their reasoning makes no sense within your worldview and appears to be making reasoning errors or bad faith arguments all the time. And so a lot of attention is spent on the arguments, rather than on the premises and framings. It’s more productive to focus on making the discussion mutually intelligible, with everyone learning towards passing everyone else’s ideological Turing test. Actually passing is unimportant, but learning towards that makes talking past each other less of a problem, and cruxes start emerging.
I think that most unpopular extreme ideas have good simple counterarguments. E.g. for Marxism it’s that it whenever people attempt it, this leads to famines and various extravagant atrocities. Of course, “real Marxism hasn’t been tried” is the go-to counter-counterargument, but even if you are a true believer, it should give you pause that it has been very difficult to implement in practice, and it’s reasonable for people to be critical by default because of those repeated horrible failures.
The clear AI implication I addressed elsewhere.
I actually agree with this for Marxism, but I generally think these are the exceptions, not the rule, and bad ideas like these will often require more subtle counterarguments that the general public won’t notice, so they fall back onto bad criticisms, and it’s here where you need to be careful about not updating based on the counter-arguments being terrible.
And I think AI is exactly such a case, where conditional on AI doom being wrong, it will be for reasons that the general public mostly won’t know/care to say, and will still give bad arguments against AI doom.
This also applies to AI optimism to a lesser extent.
Also, you haven’t linked to your comment properly, when I notice the link it goes to the post rather than your comments.
Most people are clueless about AI doom, but they have always been clueless about approximately everything throughout history, and get by through having alternative epistemic strategies of delegating sense-making and decision-making to supposed experts.
Supposed experts clearly don’t take AI doom seriously, considering that many of them are doing their best to race as fast as possible, therefore people don’t either, an attitude that seems entirely reasonable to me.
Thank you, fixed.
I agree with this, and this is basically why I was saying that you shouldn’t update towards your view being correct based on the general public making bad arguments, because this is the first step towards false beliefs based on selection effects.
It was in a sense part of my motivation for posting this at all.
This is half correct on how much experts take AI doom seriously, and some experts like Yoshua Bengio or Geoffrey Hinton do take AI doom seriously, and I agree that their attitude is reasonable (though for different reasons than you would say)
My point is that “experts disagree with each other, therefore we’re justified in not taking it seriously” is a good argument, and this is what people mainly believe. If they instead offer bad object-level arguments, then sure, dismissing those is fine and proper.
I agree that their attitude is reasonable, conditional on superintelligence being achievable in the foreseeable future. I personally think this is unlikely, but I’m far from certain.
I was referring to the general public here.