I think the key characteristic of motivated reasoning is that you ignore some knowledge or model that you would ordinarily employ while under less pressure.
A pretty standard definition of motivated reasoning is that it is reasoning that is actively working towards reaching a certain preferred conclusion,
Quoting Googles AI overview (which is generally pretty terrible, but suffices here),
“Motivated reasoning is the psychological tendency to process information in a biased way, seeking out evidence that supports what we want to be true (our beliefs, desires, identity) while dismissing contradictory facts, often unconsciously, to avoid discomfort or maintain a positive self-image.”
It doesn’t require that you already have the knowledge or model, if you would otherwise acquire it if you weren’t trying to reach a certain conclusion. Failure to learn new things is far more central, because if you already have well integrated models it becomes hard to form the broken intentions in the first place.
If you were to choose to intentionally output only 50J, while predicting that this would somehow reach the desired temperature (contrary to the model you regularly employ in more tractable situations), then I would consider that a central example of motivated reasoning.
I think there are a lot of missing pieces in your picture here. How do you operationalize “intentionally”, for one? Like, how do you actually test whether a system was “intentional” or “just did a thing”? If a system can’t put out more than 50j, in what sense is 50j the intention and not 100 or “more” or something else?
Rather, you seem to be describing a reaction where you try to output 100J, meaning you are choosing an action that is actually powerful enough to accomplish your goal, but which will have undesirable side-effects.
Well, not necessarily, which is why I said “and maybe”. If I program in a maximum pulse width, the controller upstream doesn’t know about it. It puts out a new value, which maybe would or maybe wouldn’t be enough, but it can’t know. All it knows is that it didn’t work this time, and it’s not updating on the possibility that maybe failing the last twenty times in a row means the temperature won’t actually reach the setpoint.
I suppose if your heating element is in fact incapable of outputting 100J (even if you allow side-effects), and you are aware of this limitation, and you choose to ask for 100J anyway, while expecting this to somehow generate 100J (directly contra the knowledge we just assumed you have), then that would count as motivated reasoning.
That is far closer to the point. The controller makes motions that would work under its model of the world… in expectation, without any perceived guarantee of this being reality… and in reality that isn’t happening.
The problem now is in the interaction between the meta level and the object level.
On the object level, the controller is still forming its conclusions of what will happen based on what it wants to happen. This is definitionally motivated cognition in a sense, but it’s only problematic when the controller fails. The object level controller itself, by definition of “object level”, is in the business of updating reality not its model of reality. The problematic sense comes in when the meta level algorithm that oversees the object level controller chooses not to deliver all the information to the object level controller because that would cause the controller to stop trying, and the meta level algorithm doesn’t think that’s a good idea.
Let’s look at the case of the coach saying “You gotta BELIEVE!”. This is an explicit endorsement of motivated reasoning. The motivational frame he’s operating in is that you expect to win, figure out what you gotta do to get there, and then do the things. The problem with giving this object level controller full info is that “Oh, I’m not gonna win” is a conclusion it might reach, and then what actions will it output? If you’re not gonna win, what’s it matter what you do next? If full effort is costly, you’re not going to do it when you’re not going to win anyway.
When you shift from controlling towards “win” to controlling towards the things that maximize chances of winning, then “I’m not gonna win though” becomes entirely irrelevant. Not something you have to hide from the controller, just something that doesn’t affect decision making. “Okay so I’m gonna lose. I’m still going to put in 100% effort because I’m going to be the person who never loses unnecessarily”.
The motivated reasoning, and explicit endorsement of such, comes from the fact that being fully honest can cause stupid reactions, and if you don’t know how to use that additional information well, updating on it can result in stupider actions (from the perspective of the meta layer). Same thing with “No, this dress doesn’t make your ass look fat honey”/”She’s just gonna get upset. Why would I upset her?” coming from a person who doesn’t know how to orient to difficult realities.
because you are inferring the controller’s “expectations” purely from its actions, and this type of inference doesn’t allow you to distinguish between “the controller is unaware that its heating element can’t output 100J” from “the controller is aware, but choosing to pretend otherwise”.
Oh, no, you can definitely distinguish. The test is “What happens when you point at it?”. Do they happily take the correction, or do they get grumpy at you and take not-fully-effective actions to avoid updating on what you’re pointing at? Theoretically it can get tricky, but the pretense is rarely convincing, in practice.
With simple bimetallic thermostat, it’s pretty clear from inspection that there’s just no place to put this information, so it’s structurally impossible for it to be aware of anything else. Alternatively, if you dig through the code and find a line “while output>maxoutput, temp—”, you can run the debugger and watch the temperature estimate get bullshitted as necessary in order to maintain the expectation.
Meta-level feedback:
I can’t help but notice that the account you’re offering is fairly presumptuous, makes quite a few uncharitable assumptions, and doesn’t show a lot of interest in learning something like “Oh, the relevance of the response time thing wasn’t clear? I’ll try again from another angle”. It’d be a lot easier to take your feedback the way you want it taken if you tried first to make sure you weren’t just missing things that I’d be happy to explain.
If you’re wed to that framing then I agree it’s probably a waste of your time continue. If you’re interested in receiving meta level feedback yourself, I can explain how I see things, why, and we can find out together what holds up and what doesn’t.
Amusingly, this would require neither of us controlling towards “being right” and instead controlling towards the humility/honesty/meta-perspective-taking/etc that generates rightness. Might be an interesting demonstration of the thing I’m trying to convey, if you want to try that.
Also, sorry if it’s gotten long again. I’m pretty skeptical that a shorter solution exists at all, but if it does I certainly can’t find it. Heck, I’d be pleasantly surprised if it all made sense at this length.
A pretty standard definition of motivated reasoning is that it is reasoning that is actively working towards reaching a certain preferred conclusion,
Quoting Googles AI overview (which is generally pretty terrible, but suffices here),
“Motivated reasoning is the psychological tendency to process information in a biased way, seeking out evidence that supports what we want to be true (our beliefs, desires, identity) while dismissing contradictory facts, often unconsciously, to avoid discomfort or maintain a positive self-image.”
It doesn’t require that you already have the knowledge or model, if you would otherwise acquire it if you weren’t trying to reach a certain conclusion. Failure to learn new things is far more central, because if you already have well integrated models it becomes hard to form the broken intentions in the first place.
I think there are a lot of missing pieces in your picture here. How do you operationalize “intentionally”, for one? Like, how do you actually test whether a system was “intentional” or “just did a thing”? If a system can’t put out more than 50j, in what sense is 50j the intention and not 100 or “more” or something else?
Well, not necessarily, which is why I said “and maybe”. If I program in a maximum pulse width, the controller upstream doesn’t know about it. It puts out a new value, which maybe would or maybe wouldn’t be enough, but it can’t know. All it knows is that it didn’t work this time, and it’s not updating on the possibility that maybe failing the last twenty times in a row means the temperature won’t actually reach the setpoint.
That is far closer to the point. The controller makes motions that would work under its model of the world… in expectation, without any perceived guarantee of this being reality… and in reality that isn’t happening.
The problem now is in the interaction between the meta level and the object level.
On the object level, the controller is still forming its conclusions of what will happen based on what it wants to happen. This is definitionally motivated cognition in a sense, but it’s only problematic when the controller fails. The object level controller itself, by definition of “object level”, is in the business of updating reality not its model of reality. The problematic sense comes in when the meta level algorithm that oversees the object level controller chooses not to deliver all the information to the object level controller because that would cause the controller to stop trying, and the meta level algorithm doesn’t think that’s a good idea.
Let’s look at the case of the coach saying “You gotta BELIEVE!”. This is an explicit endorsement of motivated reasoning. The motivational frame he’s operating in is that you expect to win, figure out what you gotta do to get there, and then do the things. The problem with giving this object level controller full info is that “Oh, I’m not gonna win” is a conclusion it might reach, and then what actions will it output? If you’re not gonna win, what’s it matter what you do next? If full effort is costly, you’re not going to do it when you’re not going to win anyway.
When you shift from controlling towards “win” to controlling towards the things that maximize chances of winning, then “I’m not gonna win though” becomes entirely irrelevant. Not something you have to hide from the controller, just something that doesn’t affect decision making. “Okay so I’m gonna lose. I’m still going to put in 100% effort because I’m going to be the person who never loses unnecessarily”.
The motivated reasoning, and explicit endorsement of such, comes from the fact that being fully honest can cause stupid reactions, and if you don’t know how to use that additional information well, updating on it can result in stupider actions (from the perspective of the meta layer). Same thing with “No, this dress doesn’t make your ass look fat honey”/”She’s just gonna get upset. Why would I upset her?” coming from a person who doesn’t know how to orient to difficult realities.
Oh, no, you can definitely distinguish. The test is “What happens when you point at it?”. Do they happily take the correction, or do they get grumpy at you and take not-fully-effective actions to avoid updating on what you’re pointing at? Theoretically it can get tricky, but the pretense is rarely convincing, in practice.
With simple bimetallic thermostat, it’s pretty clear from inspection that there’s just no place to put this information, so it’s structurally impossible for it to be aware of anything else. Alternatively, if you dig through the code and find a line “while output>maxoutput, temp—”, you can run the debugger and watch the temperature estimate get bullshitted as necessary in order to maintain the expectation.
I can’t help but notice that the account you’re offering is fairly presumptuous, makes quite a few uncharitable assumptions, and doesn’t show a lot of interest in learning something like “Oh, the relevance of the response time thing wasn’t clear? I’ll try again from another angle”. It’d be a lot easier to take your feedback the way you want it taken if you tried first to make sure you weren’t just missing things that I’d be happy to explain.
If you’re wed to that framing then I agree it’s probably a waste of your time continue. If you’re interested in receiving meta level feedback yourself, I can explain how I see things, why, and we can find out together what holds up and what doesn’t.
Amusingly, this would require neither of us controlling towards “being right” and instead controlling towards the humility/honesty/meta-perspective-taking/etc that generates rightness. Might be an interesting demonstration of the thing I’m trying to convey, if you want to try that.
Also, sorry if it’s gotten long again. I’m pretty skeptical that a shorter solution exists at all, but if it does I certainly can’t find it. Heck, I’d be pleasantly surprised if it all made sense at this length.
Sure, give me meta-level feedback.