LessWrong has a reputation for being a place where dry and earnest people write dry and earnest essays with titles like “Don’t Believe Wrong Things”. A casual visitor wouldn’t expect it to host lively discussions of prophets, of wizards, and of achieving enlightenment. And yet, each of the above links does lead to LessWrong, and each post (including mine) has more than a hundred comments.
The discussion often turns to a debate that rages eternal in the rationalist community: correctness vs. usefulness. Rationality is about having true beliefs, we are told, but rationalists should also win. Winning, aka instrumental rationality, sure sounds a lot more fun than just believing true things (epistemic rationality). People are tempted to consider it as the primary goal of rationality, with the pursuit of truth being secondary.
Mentions of the “useful but incorrect”, which is how I see Jordan Peterson, invite comments like this:
A correct epistemological process is likely to assign very low likelihood to the proposition of Christianity being true at some point. Even if Christianity is true, most Christians don’t have good epistemics behind their Christianity; so if there exists an epistemically justifiable argument for ‘being a Christian’, our hypothetical cradle-Christian rationalist is likely to reach the necessary epistemic skill level to see through the Christian apologetics he’s inherited before he discovers it.
At which point he starts sleeping in on Sundays; loses the social capital he’s accumulated through church; has a much harder time fitting in with Christian social groups; and cascades updates in ways that are, given the social realities of the United States and similar countries, likely to draw him toward other movements and behavior patterns, some of which are even more harmful than most denominations of Christianity, and away from the anthropological accumulations that correlate with Christianity, some of which may be harmful but some of which may be protecting against harms that aren’t obvious even to those with good epistemics. Oops! Is our rationalist winning?
[…]
epistemic rationality is important because it’s important for instrumental rationality. But the thing we’re interested in is instrumental rationality, not epistemic rationality. If the instrumental benefits of being a Christian outweigh the instrumental harms of being a Christian, it’s instrumentally rational to be a Christian. If Christianity is false and it’s instrumentally rational to be a Christian, epistemic rationality conflicts with instrumental rationality.
Well, it’s time for a dry and earnest essay (probably overdue after last week’s grapefruits) on the question of instrumental vs. epistemic rationality. I am not breaking any ground that wasn’t previously covered in the Sequences etc., but I believe that this exercise is justified in the spirit of non-expert explanation.
I will attempt to:
Dissolve a lot of the dichotomy between “useful” and “correct”, via some examples that use “wrong” wrong.
Of the dichotomy that remains, position myself firmly on the correct side of the debate.
Suggest that convincing yourself of something wrong is, in fact, possible and should be guarded vigilantly against.
Say some more in praise of fake frameworks, and what they mean if they don’t mean “believing in false things”.
Wrong and Less Wrong
What does “truth” mean, for example in the definition of epistemic rationality as “the pursuit of true beliefs about the world”? I think that a lot of the apparent conflict between the “useful” and “true” stems from confusion about the latter word that isn’t merely semantic. As exemplars of this confusion, I will use Brian Lui’s posts: wrong models are good, correct models are bad, and useful models are better than correct models.
I have chosen Brian as a foil because:
We actually disagree, but both do so in good faith.
I asked him if I could, and he said OK.
Here are some examples that Brian uses:
Correct Models
Schrödinger’s model | Calorie-in-calorie-out | Big 5 personality | Spheric Earth
Useful Model
Bohr’s atomic model | Focus on satiety | MBTI | Flat Earth
You may be familiar with Asimov’s quote:
“When people thought the earth was flat, they were wrong. When people thought the earth was spherical, they were wrong. But if you think that thinking the earth is spherical is just as wrong as thinking the earth is flat, then your view is wronger than both of them put together.”
People often overlook the broader context of the quote. Asimov makes the point that Flat Earth is actuallya very good model. Other models could posit an Earth with infinitely tall mountains or bottomless trenches, or perhaps an Earth tilted in such a way that walking north-west would always be uphill. A flat Earth model, built on empiricism and logic, is quite an achievement:
Perhaps it was the appearance of the plain that persuaded the clever Sumerians to accept the generalization that the earth was flat; that if you somehow evened out all the elevations and depressions, you would be left with flatness. Contributing to the notion may have been the fact that stretches of water (ponds and lakes) looked pretty flat on quiet days.
A model is correct or not in the context of a specific question asked of it, such as “Will I arrive back home from the east if I keep sailing west?” The flat Earth model was perfectly fine until that question was asked, and the first transoceanic voyages took place more than 1,000 after Eratosthenes calculated the spherical Earth’s radius with precision.
But it’s not just the “wrong” models that are true, the opposite is also the case, as famously noticed by George Box. The Earth’s shape isn’t a sphere. It’s not even a geoid, it changes moment by moment with the tides, plate tectonics, and ants building anthills. Brian’s division of models into the correct and incorrect starts to seem somewhat arbitrary, so what is it based on?
Brian considers the Big 5 personality model to more “correct” and “scientific” because it was created using factor analysis, while Myers-Briggs is based on Jung’s conceptual theory. But the trappings of science don’t make a theory true, particularly when the science in question has a fraught relationship with the truth. How “scientific” a process was used to generate a model can correlate with its truthfulness, but as a definition it seems to miss the mark entirely.
Rationalists usually measure the truth of a model by the rent it payswhen it collides with reality. Neither MBTI nor Big 5 does a whole lot of useful prediction, and they’re not even as fun as the MTG color system. On the other hand, Bohr’s atomic model works for most questions of basic chemistry and even the photoelectric effect.
A model is wrong not because it is not precisely quantified (like satiety), or because it wasn’t published in a science journal (like MBTI), or because it has been superseded by a more reductionist model (like Bohr’s atom). It is wrong when it predicts things that don’t happen or prohibits things that do.
When a model’s predictions and prohibitions line up with observable reality, the model is true. When those predictions are easy to make and check, it is useful. Calorie-in-calorie-out isn’t very useful on the question of successful dieting because it is so difficult for people to just change their caloric balance as an immediate action. This difficulty doesn’t make this model any more or less correct, it just means that it’s hard to establish its correctness from seeing whether people who try to count calories lose weight or not. In this view truth and usefulness are almost orthogonal: truth is a precondition for usefulness, while some models are so wrong that they are worse than useless.
Jesus and Gandhi
Usefulness, in the sense of beliefs paying rent, is a narrower concept than winning, e.g., making money to pay your actual rent. The comment about the lapsed Christian I quoted talks about instrumental rationality as the pursuit of actually winning in life. So, is the rejection of Christ epistemically rational but instrumentally irrational?
First of all, I think that the main mistake the hypothetical apostate is making is a bucket error. In his mind, there is a single variable labeled “Christianity” which contains a boolean value: True or False. This single variable serves as an answer to many distinct questions, such as:
Did Jesus die for my sins?
Should I go to church on Sunday?
Should I be nice to my Christian friends?
There is no reason why all three questions must have the same answer, as demonstrated by my closet-atheist friend who lives in an Orthodox Jewish community. The rent in the Jewish part of Brooklyn is pretty cheap (winning!) and doesn’t depend on one’s beliefs about revelation. Living a double life is not ideal, and it is somewhat harder to fit in a religious community if you’re a non-believer. But carelessly propagating new beliefs before sorting out the buckets in one’s head is much more dangerous than zoning out during prayer times. Keeping behaviors that correlate with a false belief is very different from installing new beliefs to change one’s behavior.
Information hazards are also a thing. There are many real things that we wish other people wouldn’t know, and some things that we wouldn’t want to learn ourselves. But avoiding true but dangerous knowledge is also very different than hunting false beliefs.
With that said, what if hunting and installing false beliefs is actually justified? A friend of mine who’s a big fan of Jordan Peterson is joking-not-joking about converting to Christianity. If Christianity provides one with friends, meaning, and protection from harmful ideologies, isn’t it instrumentally rational to convert?
There’s a word for this sort of bargain: Faustian. One should always imagine this spoken by someone with reddish skin, twisty horns, and an expensive suit. I offer you all this, and all I want in return is a tiny bit of epistemic rationality. What’s it even worth to you?
Epistemic rationality is worth a lot.
It takes a lot of epistemic rationality to tease apart causation from the mere correlation of religion with its benefits. Perhaps a Christian’s community likes him because consistent beliefs make a person predictable; this benefit wouldn’t extend to a fresh convert. As for meaning and protection from adverse memes, are those provided by Jesus or by the community itself? Or by some confounder like age or geography?
A person discerning enough on matters of friendship to judge whether it is the cause or the effect of Christian belief probably understands friendship well enough to make friends with or without converting. I help run a weekly meetup of rationalists in New York. We think a lot about building an active community, and we implement this in practice. We may not provide the full spiritual package of a church, but we also don’t demand a steep price from our members: neither in money, nor in effort, nor in dogma.
Perhaps converting is the instrumentally optimal thing to do for a young rationalist, but it would require heroic epistemic rationality to know that it is so. And once you have converted, that epistemic rationality is gone forever, along with the ability to reason well about such trade-offs in the future. If you discover a new religion tomorrow that offers ten times the benefits of Christianity, it would be too late: your new belief in the truth of Christianity will prevent you from even considering the option of reconverting to the new religion.
This argument is colloquially known as The Legend of Murder-Gandhi. Should Gandhi, who abhors violence, take a pill that makes him 99% as reluctant to commit murder for a million dollars? No, because 99%-pacifist Gandhi will not hesitate to take another pill and go to 98%, and then down to 97%, and to 90%,
and so on until he’s rampaging through the streets of Delhi, killing everything in sight.
An exception could be made if Gandhi had a way to commit himself to stopping at 95% pacifism; that’s still pacifist enough that he doesn’t really need to worry about acting violently, yet $5 million richer.
But epistemic rationality is a higher-level skill than mere pacifism. It’s the skill that’s necessary not only to assess a single trade-off, but also to understand the dangers of slippery slopes, and the benefits of pre-commitments, and the need for Functional Decision Theory in a world full of Newcomblike problems. Gandhi who’s perfectly pacifist but doesn’t understand Schelling fences will take the first pill, and all his pacifism will be for naught.
Do you think you have enough epistemic rationality to determine when it’s really worth sacrificing epistemic rationality for something else? Better to keep increasing your epistemic rationality, just to be sure.
Flat Moon Society
Is this a moot point, though? It’s not like you can make yourself go to sleep an atheist and wake up a devout Christian tomorrow. Eliezer wrote a whole sequence on the inability to self-deceive:
We do not have such direct control over our beliefs. You cannot make yourself believe the sky is green by an act of will. You might be able to believe you believed it—though I have just made that more difficult for you by pointing out the difference. (You’re welcome!) You might even believe you were happy and self-deceived; but you would not in fact be happy and self-deceived.
[…] You can’t know the consequences of being biased, until you have already debiased yourself. And then it is too late for self-deception. The other alternative is to choose blindly to remain biased, without any clear idea of the consequences. This is not second-order rationality. It is willful stupidity.
He gives an example of very peculiar Orthodox Jew:
When this woman was in high school, she thought she was an atheist. But she decided, at that time, that she should act as if she believed in God. And then—she told me earnestly—over time, she came to really believe in God. So far as I can tell, she is completely wrong about that. Always throughout our conversation, she said, over and over, “I believe in God”, never once, “There is a God.” When I asked her why she was religious, she never once talked about the consequences of God existing, only about the consequences of believing in God. Never, “God will help me”, always, “my belief in God helps me”. When I put to her, “Someone who just wanted the truth and looked at our universe would not even invent God as a hypothesis,” she agreed outright.
She hasn’t actually deceived herself into believing that God exists or that the Jewish religion is true. Not even close, so far as I can tell.
On the other hand, I think she really does believe she has deceived herself.
But eventually, he admits that believing you won’t self-deceive is also somewhat of a self-fulfilling prophecy:
It may be wise to go around deliberately repeating “I can’t get away with double-thinking! Deep down, I’ll know it’s not true! If I know my map has no reason to be correlated with the territory, that means I don’t believe it!”
Because that way—if you’re ever tempted to try—the thoughts “But I know this isn’t really true!” and “I can’t fool myself!” will always rise readily to mind; and that way, you will indeed be less likely to fool yourself successfully. You’re more likely to get, on a gut level, that telling yourself X doesn’t make X true: and therefore, really truly not-X.
To me the sequence’s message is “don’t do it!” rather than “it’s impossible!”. If self-deception were impossible, there would be no need for injunctions against it.
Self-deception definitely isn’t easy. A good friend of mine told me about two guys he knows who are aspiring flat-Earthers. Out of the pure joy of contrarianism, the two have spent countless hours watching flat-Earth apologia on YouTube. So far their yearning for globeless epiphany hasn’t been answered, although they aren’t giving up.
A coworker of mine feels that every person should believe in at least one crazy conspiracy theory, and so he says that he convinced himself that the moon landing was faked. It’s hard to tell if he fully believes it, but he probably believes it somewhat. His actual beliefs about NASA have changed, not just his beliefs-in-self-deception. Perhaps earlier in life, he would have bet that the moon landing was staged in a movie studio at million-to-one odds, and now he’ll take that bet at 100:1.
He is certainly less likely to discount the other opinions of moon landing-skeptics, which leaves him a lot more vulnerable to being convinced of bullshit in the future. And the mere belief-in-belief is still a wrong belief that was created in his mind ex-nihilo. My colleague clearly sacrificed some amount of epistemic rationality, although it’s unclear what he got in return.
Self-deception works like deception. False beliefs sneak into your brain the same way a grapefruit does.
First, we hear something stated as fact: the moon landing was staged. Our brain’s immediate reaction on a neurological level to a new piece of information is to believe it. Only when propagating the information shows it to be in conflict with prior beliefs is it discarded. But nothing can ever be discarded entirely by our brains, and a small trace remains.
We come across the same information a few more times. Now, the brain recognizes it as familiar, which means that it anchors itself deeper into the brain even if it is disbelieved every time. The traces accumulate. Was the footage of the moon landing really all it seemed?
Perhaps we associate a positive feeling with the belief. Wouldn’t it be cool if the Appolo missions never happened? This means that I can still be the first human on the moon!
Even if we still don’t believe the original lie when questioning it directly, it still occupies some territory in our head. Adjacent beliefs get reinforced through confirmation bias, which in turn reinforces the original lie. If the “landing” was really shot on the moon, why was the flag rippling in the wind? Wait, is the flag actually rippling? We don’t remember, it’s not like we watch moon landing footage every day. But now we believe that the flag was rippling, which reinforces the belief that the moon landing was fake.
We forget where we initially learned the information from. Even if the original claim about the moon fakery was presented as untrue and immediately debunked, we will just remember that we heard somewhere that it was all an elaborate production to fool the Russians. We recall that we used to be really skeptical of the claim once, but it sure feels like a lot of evidence has been pointing that way recently…
It is easiest to break this chain on step 1 – avoid putting trash into your brain. As an example, I will never read the Trump exposé Fire and Fury under any circumstances, and implore my friends to do the same. Practically everyone agrees that the book has ten rumors and made up stories for every single verifiable fact, but if you read the book, you don’t know which is which. If you’re the kind of person who’s already inclined to believe anything and everything about Donald Trump, reading the book will inevitably make you stupider and less informed about the president. And this “kind of person” apparently includes most of the country, because no parody of Fire and Furyhas been too outlandish to be believed.
Take the Glasses Off
So, what are “fake frameworks” and what do they have to do with all of this?
I use a lot of fake frameworks — that is, ways of seeing the world that are probably or obviously wrong in some important way. […] Assume the intuition is wrong. It’s fake. And then use it anyway.
It almost sounds as if Val is saying that we should believe in wrong things, but I don’t think that’s the case. Here’s the case.
First of all, you should use a safety mechanism when dealing with fake frameworks: sandboxing. This means holding the belief in a separate place where it doesn’t propagate.
This is why I talk about wearing a “Peterson mask”, or having Peterson as a voice on your shoulder. The goal is to generate answers to questions like “What would Peterson tell me to do here? And how would Scott Alexander respond?” rather than literally replacing your own beliefs with someone else’s. Answering those questions does require thinking as Peterson for a while, but you can build scaffolding that prevents that mode of thinking from taking over.
But sandboxing is secondary to the main point of fake frameworks: they’re not about believing new things, they’re about un-believing things.
A lot of fake frameworks deal with the behavior of large numbers of people: coordination problems are an ancient hungry demon, the social web forces people into playing roles, Facebook is out to get you. In what sense is Facebook out to get you? Facebook is thousands of employees and millions of shareholders pursuing their own interest, not a unified agent with desires.
But neither is a person.
People minds are made up of a multitude of independent processes, conscious and unconscious, each influencing our interactions with the world. Our single-minded pursuit of genetic fitness has shattered into a thousand shards of desire. Insofar as we have strategic goals such as being out to get someone, we are constantly distracted from them and constantly changing them.
The insight of fake frameworks is that every framework you use is fake, especially when talking about complicated things like people and societies. “Society” and “person” themselves aren’t ontologically basic entities, just useful abstractions. Useful, but not 100% true.
And yet, you have to interact with people and societies every day. You can’t do it without some framework of thinking about people; a cocktail party isn’t navigable on the level of quarks or molecules or cells. You have to see human interaction through one pair of glasses or another. The glasses you look through impose some meaning on the raw data of moving shapes and mouth sounds, but that meaning is “fake”: it’s part of the map, not the territory.
Once you realize that you’re wearing glasses, it’s hard to forget that fact. You can now safely take the glasses off and replace them with another pair, without confusing what you see through the lenses with what exists on a fundamental level. The process is gradual, peeling away layer after layer of immutable facts that turned out to be interpretations. Every time a layer is peeled away, you have more freedom to play with new frameworks of interpretation to replace it.
If you can stand one more visual-based metaphor, the skill of removing glass is also called Looking. This art is hard and long and I’m only a novice in it, but I have a general sense of the direction of progress. There seems to be a generalizable skill of Looking and playing with frameworks, as well as domain-specific understanding that is required for Looking in different contexts. Deep curiosity is needed, and also relinquishment. It often takes an oblique approach rather than brute force. For example, people report the illusion of a coherent “self” being dispelled by such varied methods as meditation, falling in love, taking LSD, and studying philosophy.
Finally, while I can’t claim the benefits that others can, I think that Looking offers real protection against being infected with wrong beliefs. Looking is internalizing that some of your beliefs about the world are actually interpretations you impose on it. False interpretations are much easier to critically examine and detach from than false beliefs. You end up believing fewer wrong things about the world simply because you believe fewer things about the world.
And if Looking seems beyond reach, believing fewer wrong things is always a good place to start.
Don’t Believe Wrong Things
This is cross-posted from Putanumonit.com, you can jump in the discussion in either place.
LessWrong has a reputation for being a place where dry and earnest people write dry and earnest essays with titles like “Don’t Believe Wrong Things”. A casual visitor wouldn’t expect it to host lively discussions of prophets, of wizards, and of achieving enlightenment. And yet, each of the above links does lead to LessWrong, and each post (including mine) has more than a hundred comments.
The discussion often turns to a debate that rages eternal in the rationalist community: correctness vs. usefulness. Rationality is about having true beliefs, we are told, but rationalists should also win. Winning, aka instrumental rationality, sure sounds a lot more fun than just believing true things (epistemic rationality). People are tempted to consider it as the primary goal of rationality, with the pursuit of truth being secondary.
Mentions of the “useful but incorrect”, which is how I see Jordan Peterson, invite comments like this:
Well, it’s time for a dry and earnest essay (probably overdue after last week’s grapefruits) on the question of instrumental vs. epistemic rationality. I am not breaking any ground that wasn’t previously covered in the Sequences etc., but I believe that this exercise is justified in the spirit of non-expert explanation.
I will attempt to:
Dissolve a lot of the dichotomy between “useful” and “correct”, via some examples that use “wrong” wrong.
Of the dichotomy that remains, position myself firmly on the correct side of the debate.
Suggest that convincing yourself of something wrong is, in fact, possible and should be guarded vigilantly against.
Say some more in praise of fake frameworks, and what they mean if they don’t mean “believing in false things”.
Wrong and Less Wrong
What does “truth” mean, for example in the definition of epistemic rationality as “the pursuit of true beliefs about the world”? I think that a lot of the apparent conflict between the “useful” and “true” stems from confusion about the latter word that isn’t merely semantic. As exemplars of this confusion, I will use Brian Lui’s posts: wrong models are good, correct models are bad, and useful models are better than correct models.
I have chosen Brian as a foil because:
We actually disagree, but both do so in good faith.
I asked him if I could, and he said OK.
Here are some examples that Brian uses:
Correct Models
Schrödinger’s model | Calorie-in-calorie-out | Big 5 personality | Spheric Earth
Useful Model
Bohr’s atomic model | Focus on satiety | MBTI | Flat Earth
You may be familiar with Asimov’s quote:
People often overlook the broader context of the quote. Asimov makes the point that Flat Earth is actually a very good model. Other models could posit an Earth with infinitely tall mountains or bottomless trenches, or perhaps an Earth tilted in such a way that walking north-west would always be uphill. A flat Earth model, built on empiricism and logic, is quite an achievement:
A model is correct or not in the context of a specific question asked of it, such as “Will I arrive back home from the east if I keep sailing west?” The flat Earth model was perfectly fine until that question was asked, and the first transoceanic voyages took place more than 1,000 after Eratosthenes calculated the spherical Earth’s radius with precision.
But it’s not just the “wrong” models that are true, the opposite is also the case, as famously noticed by George Box. The Earth’s shape isn’t a sphere. It’s not even a geoid, it changes moment by moment with the tides, plate tectonics, and ants building anthills. Brian’s division of models into the correct and incorrect starts to seem somewhat arbitrary, so what is it based on?
Brian considers the Big 5 personality model to more “correct” and “scientific” because it was created using factor analysis, while Myers-Briggs is based on Jung’s conceptual theory. But the trappings of science don’t make a theory true, particularly when the science in question has a fraught relationship with the truth. How “scientific” a process was used to generate a model can correlate with its truthfulness, but as a definition it seems to miss the mark entirely.
Rationalists usually measure the truth of a model by the rent it payswhen it collides with reality. Neither MBTI nor Big 5 does a whole lot of useful prediction, and they’re not even as fun as the MTG color system. On the other hand, Bohr’s atomic model works for most questions of basic chemistry and even the photoelectric effect.
A model is wrong not because it is not precisely quantified (like satiety), or because it wasn’t published in a science journal (like MBTI), or because it has been superseded by a more reductionist model (like Bohr’s atom). It is wrong when it predicts things that don’t happen or prohibits things that do.
When a model’s predictions and prohibitions line up with observable reality, the model is true. When those predictions are easy to make and check, it is useful. Calorie-in-calorie-out isn’t very useful on the question of successful dieting because it is so difficult for people to just change their caloric balance as an immediate action. This difficulty doesn’t make this model any more or less correct, it just means that it’s hard to establish its correctness from seeing whether people who try to count calories lose weight or not. In this view truth and usefulness are almost orthogonal: truth is a precondition for usefulness, while some models are so wrong that they are worse than useless.
Jesus and Gandhi
Usefulness, in the sense of beliefs paying rent, is a narrower concept than winning, e.g., making money to pay your actual rent. The comment about the lapsed Christian I quoted talks about instrumental rationality as the pursuit of actually winning in life. So, is the rejection of Christ epistemically rational but instrumentally irrational?
First of all, I think that the main mistake the hypothetical apostate is making is a bucket error. In his mind, there is a single variable labeled “Christianity” which contains a boolean value: True or False. This single variable serves as an answer to many distinct questions, such as:
Did Jesus die for my sins?
Should I go to church on Sunday?
Should I be nice to my Christian friends?
There is no reason why all three questions must have the same answer, as demonstrated by my closet-atheist friend who lives in an Orthodox Jewish community. The rent in the Jewish part of Brooklyn is pretty cheap (winning!) and doesn’t depend on one’s beliefs about revelation. Living a double life is not ideal, and it is somewhat harder to fit in a religious community if you’re a non-believer. But carelessly propagating new beliefs before sorting out the buckets in one’s head is much more dangerous than zoning out during prayer times. Keeping behaviors that correlate with a false belief is very different from installing new beliefs to change one’s behavior.
Information hazards are also a thing. There are many real things that we wish other people wouldn’t know, and some things that we wouldn’t want to learn ourselves. But avoiding true but dangerous knowledge is also very different than hunting false beliefs.
With that said, what if hunting and installing false beliefs is actually justified? A friend of mine who’s a big fan of Jordan Peterson is joking-not-joking about converting to Christianity. If Christianity provides one with friends, meaning, and protection from harmful ideologies, isn’t it instrumentally rational to convert?
There’s a word for this sort of bargain: Faustian. One should always imagine this spoken by someone with reddish skin, twisty horns, and an expensive suit. I offer you all this, and all I want in return is a tiny bit of epistemic rationality. What’s it even worth to you?
Epistemic rationality is worth a lot.
It takes a lot of epistemic rationality to tease apart causation from the mere correlation of religion with its benefits. Perhaps a Christian’s community likes him because consistent beliefs make a person predictable; this benefit wouldn’t extend to a fresh convert. As for meaning and protection from adverse memes, are those provided by Jesus or by the community itself? Or by some confounder like age or geography?
A person discerning enough on matters of friendship to judge whether it is the cause or the effect of Christian belief probably understands friendship well enough to make friends with or without converting. I help run a weekly meetup of rationalists in New York. We think a lot about building an active community, and we implement this in practice. We may not provide the full spiritual package of a church, but we also don’t demand a steep price from our members: neither in money, nor in effort, nor in dogma.
Perhaps converting is the instrumentally optimal thing to do for a young rationalist, but it would require heroic epistemic rationality to know that it is so. And once you have converted, that epistemic rationality is gone forever, along with the ability to reason well about such trade-offs in the future. If you discover a new religion tomorrow that offers ten times the benefits of Christianity, it would be too late: your new belief in the truth of Christianity will prevent you from even considering the option of reconverting to the new religion.
This argument is colloquially known as The Legend of Murder-Gandhi. Should Gandhi, who abhors violence, take a pill that makes him 99% as reluctant to commit murder for a million dollars? No, because 99%-pacifist Gandhi will not hesitate to take another pill and go to 98%, and then down to 97%, and to 90%,
An exception could be made if Gandhi had a way to commit himself to stopping at 95% pacifism; that’s still pacifist enough that he doesn’t really need to worry about acting violently, yet $5 million richer.
But epistemic rationality is a higher-level skill than mere pacifism. It’s the skill that’s necessary not only to assess a single trade-off, but also to understand the dangers of slippery slopes, and the benefits of pre-commitments, and the need for Functional Decision Theory in a world full of Newcomblike problems. Gandhi who’s perfectly pacifist but doesn’t understand Schelling fences will take the first pill, and all his pacifism will be for naught.
Do you think you have enough epistemic rationality to determine when it’s really worth sacrificing epistemic rationality for something else? Better to keep increasing your epistemic rationality, just to be sure.
Flat Moon Society
Is this a moot point, though? It’s not like you can make yourself go to sleep an atheist and wake up a devout Christian tomorrow. Eliezer wrote a whole sequence on the inability to self-deceive:
He gives an example of very peculiar Orthodox Jew:
But eventually, he admits that believing you won’t self-deceive is also somewhat of a self-fulfilling prophecy:
To me the sequence’s message is “don’t do it!” rather than “it’s impossible!”. If self-deception were impossible, there would be no need for injunctions against it.
Self-deception definitely isn’t easy. A good friend of mine told me about two guys he knows who are aspiring flat-Earthers. Out of the pure joy of contrarianism, the two have spent countless hours watching flat-Earth apologia on YouTube. So far their yearning for globeless epiphany hasn’t been answered, although they aren’t giving up.
A coworker of mine feels that every person should believe in at least one crazy conspiracy theory, and so he says that he convinced himself that the moon landing was faked. It’s hard to tell if he fully believes it, but he probably believes it somewhat. His actual beliefs about NASA have changed, not just his beliefs-in-self-deception. Perhaps earlier in life, he would have bet that the moon landing was staged in a movie studio at million-to-one odds, and now he’ll take that bet at 100:1.
He is certainly less likely to discount the other opinions of moon landing-skeptics, which leaves him a lot more vulnerable to being convinced of bullshit in the future. And the mere belief-in-belief is still a wrong belief that was created in his mind ex-nihilo. My colleague clearly sacrificed some amount of epistemic rationality, although it’s unclear what he got in return.
Self-deception works like deception. False beliefs sneak into your brain the same way a grapefruit does.
First, we hear something stated as fact: the moon landing was staged. Our brain’s immediate reaction on a neurological level to a new piece of information is to believe it. Only when propagating the information shows it to be in conflict with prior beliefs is it discarded. But nothing can ever be discarded entirely by our brains, and a small trace remains.
We come across the same information a few more times. Now, the brain recognizes it as familiar, which means that it anchors itself deeper into the brain even if it is disbelieved every time. The traces accumulate. Was the footage of the moon landing really all it seemed?
Perhaps we associate a positive feeling with the belief. Wouldn’t it be cool if the Appolo missions never happened? This means that I can still be the first human on the moon!
Even if we still don’t believe the original lie when questioning it directly, it still occupies some territory in our head. Adjacent beliefs get reinforced through confirmation bias, which in turn reinforces the original lie. If the “landing” was really shot on the moon, why was the flag rippling in the wind? Wait, is the flag actually rippling? We don’t remember, it’s not like we watch moon landing footage every day. But now we believe that the flag was rippling, which reinforces the belief that the moon landing was fake.
We forget where we initially learned the information from. Even if the original claim about the moon fakery was presented as untrue and immediately debunked, we will just remember that we heard somewhere that it was all an elaborate production to fool the Russians. We recall that we used to be really skeptical of the claim once, but it sure feels like a lot of evidence has been pointing that way recently…
It is easiest to break this chain on step 1 – avoid putting trash into your brain. As an example, I will never read the Trump exposé Fire and Fury under any circumstances, and implore my friends to do the same. Practically everyone agrees that the book has ten rumors and made up stories for every single verifiable fact, but if you read the book, you don’t know which is which. If you’re the kind of person who’s already inclined to believe anything and everything about Donald Trump, reading the book will inevitably make you stupider and less informed about the president. And this “kind of person” apparently includes most of the country, because no parody of Fire and Fury has been too outlandish to be believed.
Take the Glasses Off
So, what are “fake frameworks” and what do they have to do with all of this?
It almost sounds as if Val is saying that we should believe in wrong things, but I don’t think that’s the case. Here’s the case.
First of all, you should use a safety mechanism when dealing with fake frameworks: sandboxing. This means holding the belief in a separate place where it doesn’t propagate.
This is why I talk about wearing a “Peterson mask”, or having Peterson as a voice on your shoulder. The goal is to generate answers to questions like “What would Peterson tell me to do here? And how would Scott Alexander respond?” rather than literally replacing your own beliefs with someone else’s. Answering those questions does require thinking as Peterson for a while, but you can build scaffolding that prevents that mode of thinking from taking over.
But sandboxing is secondary to the main point of fake frameworks: they’re not about believing new things, they’re about un-believing things.
A lot of fake frameworks deal with the behavior of large numbers of people: coordination problems are an ancient hungry demon, the social web forces people into playing roles, Facebook is out to get you. In what sense is Facebook out to get you? Facebook is thousands of employees and millions of shareholders pursuing their own interest, not a unified agent with desires.
But neither is a person.
People minds are made up of a multitude of independent processes, conscious and unconscious, each influencing our interactions with the world. Our single-minded pursuit of genetic fitness has shattered into a thousand shards of desire. Insofar as we have strategic goals such as being out to get someone, we are constantly distracted from them and constantly changing them.
The insight of fake frameworks is that every framework you use is fake, especially when talking about complicated things like people and societies. “Society” and “person” themselves aren’t ontologically basic entities, just useful abstractions. Useful, but not 100% true.
And yet, you have to interact with people and societies every day. You can’t do it without some framework of thinking about people; a cocktail party isn’t navigable on the level of quarks or molecules or cells. You have to see human interaction through one pair of glasses or another. The glasses you look through impose some meaning on the raw data of moving shapes and mouth sounds, but that meaning is “fake”: it’s part of the map, not the territory.
Once you realize that you’re wearing glasses, it’s hard to forget that fact. You can now safely take the glasses off and replace them with another pair, without confusing what you see through the lenses with what exists on a fundamental level. The process is gradual, peeling away layer after layer of immutable facts that turned out to be interpretations. Every time a layer is peeled away, you have more freedom to play with new frameworks of interpretation to replace it.
If you can stand one more visual-based metaphor, the skill of removing glass is also called Looking. This art is hard and long and I’m only a novice in it, but I have a general sense of the direction of progress. There seems to be a generalizable skill of Looking and playing with frameworks, as well as domain-specific understanding that is required for Looking in different contexts. Deep curiosity is needed, and also relinquishment. It often takes an oblique approach rather than brute force. For example, people report the illusion of a coherent “self” being dispelled by such varied methods as meditation, falling in love, taking LSD, and studying philosophy.
Finally, while I can’t claim the benefits that others can, I think that Looking offers real protection against being infected with wrong beliefs. Looking is internalizing that some of your beliefs about the world are actually interpretations you impose on it. False interpretations are much easier to critically examine and detach from than false beliefs. You end up believing fewer wrong things about the world simply because you believe fewer things about the world.
And if Looking seems beyond reach, believing fewer wrong things is always a good place to start.