A frame to try on: If most failures of rationality are adaptively self-serving motivated reasoning, choosing to be an aspiring rationalist is basically aspiring to a kind of self-hobbling.
This is almost exactly counter to “rationality is systematized winning.”
Suppose that we’re living in a world where everyone is negotiating for their interests all the time. Almost everyone is engaged in motivated cognition that supports their interests in the ongoing negotiation, and that this causes them to do better for themselves on net. The rationalist is the guy who’s doing his best to weaken his negotiation position by “overcoming his [self-serving] biases.”
...
Of course, some domains reward rationality more than others, and some domains reward motivated reasoning more than others, and so the value of specializing in rationality depends on your goals and the domains you’re operating within.
If your primary interests are physics, or math, or philosophy, or identifying and implementing policies that are abstractly welfare-maximizing, or if you’re smart enough to make a lot of money via finance, rationality is a better strategy.
If you’re mostly interested in your personal resources or welfare, and are not making money in a technical field where you succeed by being right, you maybe mostly win by getting other people “on your side” in a thousand different ways, and so motivated reasoning is more rewarded.
And indeed, if you have the option of compartmentalizing your rationality, so that you can use it only in the domains where getting the right answer matters, without it interfering with your ability to otherwise get the benefits of motivated cognition in advocating for your interests, that would be the best of both worlds.
I’m not sure if I expect motivated reasoning to come out better on average, even in domains where you might naively expect it to. In part that’s because self-serving strategies often involve doing things other people don’t like, e.g. being deceptive, manipulative, or generally unethical, in a way that can cause long-term harm to your reputation and so long term harm to your ability to win. And I think there is significant optimization pressure on catching this kind of thing, in part for reasons similar to the ones outlined in Elephant in the Brain, i.e., that we evolved in an environment where winning that cat and mouse game was a big part of adaptive success. But also just because people don’t like being screwed, and so are on the lookout for this kind of behavior.
Also, in my imagination you’re more likely to win if you’re at least self-reflective about motivated cognition, since you can make more informed decisions that way. If you just go blindly ahead, then you’re probably failing to track a bunch of what matters, and so failing to win according to what you ultimately care about. Like, in most cases motivated reasoning spins up not just to convince other people, but to convince yourself, which means there’s a part of you that needed convincing in the first place, i.e., a part that is tracking and wanting different things. And I would guess that charging ahead without understanding those dynamics leads to worse outcomes overall? Another way to say it is that I don’t imagine a good rationalist as acting against their own interests, but more like they understand them clearly, such that they can decide what makes sense based on a fuller picture of their own mind.
I’m not sure if I expect motivated reasoning to come out better on average, even in domains where you might naively expect it to.
If it’s not adaptive, why do humans do it? Do you think it used to be adaptive in the ancestral environment, but the world has changed?
And I think there is significant optimization pressure on catching this kind of thing, in part for reasons similar to the ones outlined in Elephant in the Brain, i.e., that we evolved in an environment where winning that cat and mouse game was a big part of adaptive success. But also just because people don’t like being screwed, and so are on the lookout for this kind of behavior.
Isn’t the standard story that this is why there’s pressure for motivated reasoning instead of outright conscious deception and manipulation? The best way to fool others is to fool yourself.
There is definitely a standard story which says roughly “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”. I do not think that story stands up well under examination; when I think of standard day-to-day examples of motivated reasoning, that pattern sounds like a plausible explanation for some-but-a-lot-less-than-all of them.
For example: suppose it’s 10 pm and I’ve been playing Civ all evening. I know that I should get ready for bed now-ish. But… y’know, this turn isn’t a very natural stopping point. And it’s not that bad if I go to bed half an hour late, right? Etc. Obvious motivated reasoning. But man, that motivated reasoning sure does not seem very socially-oriented? Like, sure, you could make up a story about how I’m justifying myself to an imaginary audience or something, but it does not feel like one would have predicted the Civ example in advance from the model “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”.
Another class of examples: very often in social situations, the move which will actually get one the most points is to admit fault and apologize. And yet, instead of that, people instinctively spin a story about how they didn’t really do anything wrong. People instinctively spin that story even when it’s pretty damn obvious (if one actually stops to consider it) that apologizing would result in a better outcome for the person in question. Again, you could maybe make up some story about evolving suboptimal heuristics, but this just isn’t the behavior one would predict in advance from the model “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”.
A pattern with these examples (and many others): motivated reasoning isn’t mainly about fooling others, it’s about fooling oneself. Or at least a part of oneself. Indeed, there’s plenty of standard wisdom along those lines: “the easiest person to fool is yourself”, etc.
Here’s a model which I think much better matches real-world motivated reasoning. (Note, however, that all the above critique still stands regardless of whether this next model is correct.)
Motivated reasoning simply isn’t adaptive. Even in the ancestral environment, motivated reasoning decreased fitness. It appeared in the first place as an accidental side-effect of an overall-beneficial change in human minds relative to earlier minds, and that change was recent enough that evolution hasn’t had time to fix the anti-adaptive side effects.
There’s more than one hypothesis for what that change could be. Probably something to do with some particular functions within the mind being separated into parts, so that one part of the mind can now sometimes “cheat” by trying to trick another part of the mind, but the separation of those two functions is still overall beneficial.
An example falsifiable prediction of this model: other animals generally do not motivatedly-reason. If the relevant machinery had been around for very long, we would have expected evolution to fix the problem.
Motivated reasoning works because it eliminates the need for you to be actively deceptive, manipulative, or unethical. People’s detectors for those behaviors don’t fire, because you don’t even know you’re doing it. You think you’re being honest and ethical; but you believe the right things to make your honest and ethical behavior serve your interests. So it’s a win-win.
I think what you’re missing is the severe cognitive limitations we work under. Even those of us who have practiced analysis of complex situations and ourselves don’t have time to apply this carefully. And if we did do all of that analysis to figure out where , we don’t have the acting skills to pull off being deceptive or manipulative when it’s the best strategy.
Motivated reasoning is also just our default mode of reasoning; we mix together the vague reward from “this is probably right which is often helpful” with “I think believing this will probably get me rewards” (e.g., from people wanting to be my friend and help me.
You should expand this into a top-level post (both because it’s great, and to keep up the dream of still having a website for rationality and not just AI futurism).
I think that the reasons for motivated reasoning to emerge are more complex. Your proposed mechanism is that motivated reasoning is like having the internal CoT, and not just external speech, optimized[1] for being persuasive to others.
I also see a different mechanism. The infamous Buridan’s ass is the problem in decision theory where the difference between proposed benefits of two alternatives is far less than the cost of pinpointing the better one. In this case it’s likely rational to choose randomly according to probabilities based on imprecise estimates, then to stick to the chosen solution; said probabilities are likely close to 1⁄2and are prone to shift if new evidence appear. However, multiple such shifts require people to reassess many decisions or even waste resources trying to fix the mistakes. Motivated reasoning in a case of genuine uncertainty caused those who exhibit such behavior to arrive to a solution faster and to stick to the chosen opinion tighter, reducing waste.
A similar mechanism is the master-slave model where the slave is to demonstrate alignment and benefits from having terminal values similar to what the society prescribes. The slave believing in a false fact related to an agenda would also make more compelling cases for the agenda than the slave who is just tasked with defending it.
Yes. Rationality has two definitions, “figuring out the truth” and “doing what works”. These are often convergent but often divergent. I’ve wanted to write about this, but alignment keeps seeming more important since my timelines are short and my writing process is long :)
I get what you’re saying here (I do live in DC) but I generally think that the returns to “just being a simpler person” (simple in the complexity sense, not the euphemism-for-dumb sense) are deeply underrated.
One of my biggest lessons from being a bartender and waiter in DC is that genuinely prioritizing giving a guest the best answer on which wine to order, even if it resulted in a lower price point for that bottle, generally resulted in a higher total tip amount at the end of the night. People trust you more, at least in American society, if you’re just, like, trying to be honest and give your true best guesses.
This generalized to a hilarious degree to working in management consulting—I assert that a huge reason of why McKinsey is higher prestige than Booz Allen (I worked at both places) is that Booz Allen culture encourages a lot more self-serving behavior, and McKinsey culture encourages serving the best interests for your client, and this affects how people relate to you in a pretty fundamental way.
If most failures of rationality are adaptively self-serving motivated reasoning
I would say that most failures of rationality were adaptive in the ancestral environments, but I wouldn’t say they all count as “motivated reasoning”.
Simple example: Seeing a snake in the grass, and responding as if there is a snake in the grass, in the presence of ambiguous stimuli that have only a 10% chance of being a snake, could well result in more surviving offspring than a more nuanced, likely slower, and closer-to-correct estimation of the probability there is a snake. But this is not a result of motivated reasoning where someone is advocating for their interests, it’s just a hack that our evolved brains have for keeping us alive using minimal energy and time for computation because calories were scarce and snakes sometimes moved quickly.
My understanding is that many failures of rationality are adaptive in this way—trading off getting the right answer (in terms of the answer that will cause there to be most offspring in future generations, not in terms of the answer that would count as “winning” by the lights of the human involved, necessarily—evolution doesn’t care a whit about whether I feel like I’ve won or lost or advanced what I see as my interests, as a result of something my brain’s biased towards or away from) against energy and time costs. One thing that could be different now is, the situations where we will starve are fewer, and the time we have to think before deciding what to do is often more.
Motivated reasoning is a specific relatively small subset of the biases our brains are subject to, not the main handle for biases in general, I think?
And indeed, if you have the option of compartmentalizing your rationality
Not sure if you do? What you are describing here sounds very much like self deception. “Choosing to be Biased” is literally in the title of that article, which sounds exactly like what you are describing.
The other option instead of deceiving yourself is to only deceive others. Buy my impression so far has been that many rationalists take issue with intentional lying.
I have mostly accepted that I take this second choice in social situations where “lying”/”manipulation” is what’s expected and what everyone does subconsciously/habitually, as I think self deception would be even worse. (But I am open to suggestions if someone has a more ethical method for existing in social reality.)
you maybe mostly win by getting other people “on your side” in a thousand different ways, and so motivated reasoning is more rewarded.
This kind of deception/manipulation of others sounds exactly what you called unethical in this comment. (But maybe you were thinking of something else in that context, and I am not seeing the difference?) You basically said that manipulating other people is unethical whether someone is doing it intentionally or not.
A frame to try on: If most failures of rationality are adaptively self-serving motivated reasoning, choosing to be an aspiring rationalist is basically aspiring to a kind of self-hobbling.
This is almost exactly counter to “rationality is systematized winning.”
Suppose that we’re living in a world where everyone is negotiating for their interests all the time. Almost everyone is engaged in motivated cognition that supports their interests in the ongoing negotiation, and that this causes them to do better for themselves on net. The rationalist is the guy who’s doing his best to weaken his negotiation position by “overcoming his [self-serving] biases.”
...
Of course, some domains reward rationality more than others, and some domains reward motivated reasoning more than others, and so the value of specializing in rationality depends on your goals and the domains you’re operating within.
If your primary interests are physics, or math, or philosophy, or identifying and implementing policies that are abstractly welfare-maximizing, or if you’re smart enough to make a lot of money via finance, rationality is a better strategy.
If you’re mostly interested in your personal resources or welfare, and are not making money in a technical field where you succeed by being right, you maybe mostly win by getting other people “on your side” in a thousand different ways, and so motivated reasoning is more rewarded.
And indeed, if you have the option of compartmentalizing your rationality, so that you can use it only in the domains where getting the right answer matters, without it interfering with your ability to otherwise get the benefits of motivated cognition in advocating for your interests, that would be the best of both worlds.
I’m not sure if I expect motivated reasoning to come out better on average, even in domains where you might naively expect it to. In part that’s because self-serving strategies often involve doing things other people don’t like, e.g. being deceptive, manipulative, or generally unethical, in a way that can cause long-term harm to your reputation and so long term harm to your ability to win. And I think there is significant optimization pressure on catching this kind of thing, in part for reasons similar to the ones outlined in Elephant in the Brain, i.e., that we evolved in an environment where winning that cat and mouse game was a big part of adaptive success. But also just because people don’t like being screwed, and so are on the lookout for this kind of behavior.
Also, in my imagination you’re more likely to win if you’re at least self-reflective about motivated cognition, since you can make more informed decisions that way. If you just go blindly ahead, then you’re probably failing to track a bunch of what matters, and so failing to win according to what you ultimately care about. Like, in most cases motivated reasoning spins up not just to convince other people, but to convince yourself, which means there’s a part of you that needed convincing in the first place, i.e., a part that is tracking and wanting different things. And I would guess that charging ahead without understanding those dynamics leads to worse outcomes overall? Another way to say it is that I don’t imagine a good rationalist as acting against their own interests, but more like they understand them clearly, such that they can decide what makes sense based on a fuller picture of their own mind.
If it’s not adaptive, why do humans do it? Do you think it used to be adaptive in the ancestral environment, but the world has changed?
Isn’t the standard story that this is why there’s pressure for motivated reasoning instead of outright conscious deception and manipulation? The best way to fool others is to fool yourself.
There is definitely a standard story which says roughly “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”. I do not think that story stands up well under examination; when I think of standard day-to-day examples of motivated reasoning, that pattern sounds like a plausible explanation for some-but-a-lot-less-than-all of them.
For example: suppose it’s 10 pm and I’ve been playing Civ all evening. I know that I should get ready for bed now-ish. But… y’know, this turn isn’t a very natural stopping point. And it’s not that bad if I go to bed half an hour late, right? Etc. Obvious motivated reasoning. But man, that motivated reasoning sure does not seem very socially-oriented? Like, sure, you could make up a story about how I’m justifying myself to an imaginary audience or something, but it does not feel like one would have predicted the Civ example in advance from the model “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”.
Another class of examples: very often in social situations, the move which will actually get one the most points is to admit fault and apologize. And yet, instead of that, people instinctively spin a story about how they didn’t really do anything wrong. People instinctively spin that story even when it’s pretty damn obvious (if one actually stops to consider it) that apologizing would result in a better outcome for the person in question. Again, you could maybe make up some story about evolving suboptimal heuristics, but this just isn’t the behavior one would predict in advance from the model “motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans”.
A pattern with these examples (and many others): motivated reasoning isn’t mainly about fooling others, it’s about fooling oneself. Or at least a part of oneself. Indeed, there’s plenty of standard wisdom along those lines: “the easiest person to fool is yourself”, etc.
Here’s a model which I think much better matches real-world motivated reasoning. (Note, however, that all the above critique still stands regardless of whether this next model is correct.)
Motivated reasoning simply isn’t adaptive. Even in the ancestral environment, motivated reasoning decreased fitness. It appeared in the first place as an accidental side-effect of an overall-beneficial change in human minds relative to earlier minds, and that change was recent enough that evolution hasn’t had time to fix the anti-adaptive side effects.
There’s more than one hypothesis for what that change could be. Probably something to do with some particular functions within the mind being separated into parts, so that one part of the mind can now sometimes “cheat” by trying to trick another part of the mind, but the separation of those two functions is still overall beneficial.
An example falsifiable prediction of this model: other animals generally do not motivatedly-reason. If the relevant machinery had been around for very long, we would have expected evolution to fix the problem.
Motivated reasoning works because it eliminates the need for you to be actively deceptive, manipulative, or unethical. People’s detectors for those behaviors don’t fire, because you don’t even know you’re doing it. You think you’re being honest and ethical; but you believe the right things to make your honest and ethical behavior serve your interests. So it’s a win-win.
I think what you’re missing is the severe cognitive limitations we work under. Even those of us who have practiced analysis of complex situations and ourselves don’t have time to apply this carefully. And if we did do all of that analysis to figure out where , we don’t have the acting skills to pull off being deceptive or manipulative when it’s the best strategy.
Motivated reasoning is also just our default mode of reasoning; we mix together the vague reward from “this is probably right which is often helpful” with “I think believing this will probably get me rewards” (e.g., from people wanting to be my friend and help me.
You should expand this into a top-level post (both because it’s great, and to keep up the dream of still having a website for rationality and not just AI futurism).
I added the last paragraph, just for you.
Thanks, I hate it
I’ll think about it. I might want to develop the idea some, first.
I think that the reasons for motivated reasoning to emerge are more complex. Your proposed mechanism is that motivated reasoning is like having the internal CoT, and not just external speech, optimized[1] for being persuasive to others.
I also see a different mechanism. The infamous Buridan’s ass is the problem in decision theory where the difference between proposed benefits of two alternatives is far less than the cost of pinpointing the better one. In this case it’s likely rational to choose randomly according to probabilities based on imprecise estimates, then to stick to the chosen solution; said probabilities are likely close to 1⁄2 and are prone to shift if new evidence appear. However, multiple such shifts require people to reassess many decisions or even waste resources trying to fix the mistakes. Motivated reasoning in a case of genuine uncertainty caused those who exhibit such behavior to arrive to a solution faster and to stick to the chosen opinion tighter, reducing waste.
A similar mechanism is the master-slave model where the slave is to demonstrate alignment and benefits from having terminal values similar to what the society prescribes. The slave believing in a false fact related to an agenda would also make more compelling cases for the agenda than the slave who is just tasked with defending it.
Yes. Rationality has two definitions, “figuring out the truth” and “doing what works”. These are often convergent but often divergent. I’ve wanted to write about this, but alignment keeps seeming more important since my timelines are short and my writing process is long :)
I did write a little about my conclusions after studying motivated reasoning and cognitive biases as a major focus for some years, just as a brief answer to a question: Motivated reasoning/confirmation bias as the most important cognitive bias
I get what you’re saying here (I do live in DC) but I generally think that the returns to “just being a simpler person” (simple in the complexity sense, not the euphemism-for-dumb sense) are deeply underrated.
One of my biggest lessons from being a bartender and waiter in DC is that genuinely prioritizing giving a guest the best answer on which wine to order, even if it resulted in a lower price point for that bottle, generally resulted in a higher total tip amount at the end of the night. People trust you more, at least in American society, if you’re just, like, trying to be honest and give your true best guesses.
This generalized to a hilarious degree to working in management consulting—I assert that a huge reason of why McKinsey is higher prestige than Booz Allen (I worked at both places) is that Booz Allen culture encourages a lot more self-serving behavior, and McKinsey culture encourages serving the best interests for your client, and this affects how people relate to you in a pretty fundamental way.
i think it could be u-shaped: an initial hobbling followed by benefits from finer-grained self-control
Yeah, I think this is a plausible story.
I would say that most failures of rationality were adaptive in the ancestral environments, but I wouldn’t say they all count as “motivated reasoning”.
Simple example: Seeing a snake in the grass, and responding as if there is a snake in the grass, in the presence of ambiguous stimuli that have only a 10% chance of being a snake, could well result in more surviving offspring than a more nuanced, likely slower, and closer-to-correct estimation of the probability there is a snake. But this is not a result of motivated reasoning where someone is advocating for their interests, it’s just a hack that our evolved brains have for keeping us alive using minimal energy and time for computation because calories were scarce and snakes sometimes moved quickly.
My understanding is that many failures of rationality are adaptive in this way—trading off getting the right answer (in terms of the answer that will cause there to be most offspring in future generations, not in terms of the answer that would count as “winning” by the lights of the human involved, necessarily—evolution doesn’t care a whit about whether I feel like I’ve won or lost or advanced what I see as my interests, as a result of something my brain’s biased towards or away from) against energy and time costs. One thing that could be different now is, the situations where we will starve are fewer, and the time we have to think before deciding what to do is often more.
Motivated reasoning is a specific relatively small subset of the biases our brains are subject to, not the main handle for biases in general, I think?
Not sure if you do? What you are describing here sounds very much like self deception. “Choosing to be Biased” is literally in the title of that article, which sounds exactly like what you are describing.
The other option instead of deceiving yourself is to only deceive others. Buy my impression so far has been that many rationalists take issue with intentional lying.
I have mostly accepted that I take this second choice in social situations where “lying”/”manipulation” is what’s expected and what everyone does subconsciously/habitually, as I think self deception would be even worse. (But I am open to suggestions if someone has a more ethical method for existing in social reality.)
This kind of deception/manipulation of others sounds exactly what you called unethical in this comment. (But maybe you were thinking of something else in that context, and I am not seeing the difference?) You basically said that manipulating other people is unethical whether someone is doing it intentionally or not.