At some point, a temperature control system needs to take actions to control the temperature. Choosing the correct action depends on responding to what the temperature actually is, not what you want it to be, or what you expect it to be after you take the (not-yet-determined) correct action.
If you are picking your action based on predictions, you need to make conditional predictions based on different actions you might take, so that you can pick the action whose conditional prediction is closer to the target. And this means your conditional predictions can’t all be “it will be the target temperature”, because that wouldn’t let you differentiate good actions from bad actions.
It is possible to build an effective temperature control system that doesn’t involve predictions at all; you can precompute a strategy (like “turn heater on below X temp, turn it off above Y temp”) and program the control system to execute that strategy without it understanding how the strategy was generated, and in that case it might not have models or make predictions at all. But if you were going to rely on predictions to pick the correct action, it would be necessary to make some (conditional) predictions that are not simply “I will succeed”.
Optimal controls is something I do professionally, and the (reasonable) misconceptions you have about controls are exactly the kind that produce the (reasonable) misconceptions get people stuck with motivated reasoning.
I’ll focus on the controls first, since it’s easier to see in simpler better defined situations first, and then tie it back in to the human failures.
Choosing the correct action depends on responding to what the temperature actually is, not what you want it to be, or what you expect it to be after you take the (not-yet-determined) correct action.
So, you do have to respond to the data, obviously.
But like, the correct action also depends on what you want the temperature to be. If the jacuzzi is 100f, the correct action is different if you want it to be 101 than if you want it to be at 99.
When you actually try to build an optimal controller, ideas like “You respond to what the temperature actually is” fall apart. It takes several seconds to get a good temperature estimate from a thermometer. You read the thermometer, calculate new outputs, and change the underlying reality many times per second. By the time you’ve gathered enough data to make a decent estimate, the state changed long ago. If you’re really pushing the limits even the most recent data is out of date by the time you’ve parsed it, and this has very significant effects. This is what I spent this whole week dealing with at work, actually.
When doing optimal control you’re constantly thinking about what was your estimate in the most recent past timestep, and what it will be in the next timestep in the future. It’s rapid iteration between “I think it will be X”, “oops, lemme correct that”, “I think it will be X”. The key insight here is that this process of “oops, lemme correct that” binds future expectation to the desired value, at every future timestep.
The prediction for the next timestep will always be equal to the setpoint in an unsaturated optimal controller, exactly, because that’s what defines optimality. If you choose an output that results in 69.9f in expectation, then you could make an argument that you’re optimally controlling to 69.9f, but you’re not optimally controlling to 70f because outputting more heat would have done better by that metric.
The obvious response to this is “What if it can’t get to 70f by the next timestep, even at max power!?”, and the answer is that this would mean it’s saturated. Saturation changes things in interesting ways which I’ll return to at the end.
If you are picking your action based on predictions, you need to make conditional predictions based on different actions you might take, so that you can pick the action whose conditional prediction is closer to the target. And this means your conditional predictions can’t all be “it will be the target temperature”, because that wouldn’t let you differentiate good actions from bad actions.
You don’t need to make conditional predictions at all. Most control systems don’t. A Kalman filter coupled with an optimal control law will make unconditional predictions only, for example.
It’s true that if you try to reason the way you describe and bullshit your answers you’ll get bad results, but it doesn’t have anything to do with what I’m saying. Even if I reason “If I don’t eat next month, I’ll starve, and that’s bad”, the thing that follows is “so I’m not gonna do that”. At the end of the day, I expect to not starve, because I intend to not starve.
and program the control system to execute that strategy without it understanding how the strategy was generated,
Control systems never understand how the strategy was generated. Control systems are what do the control. They implement the strategy. Controls engineers are what do the understanding of how the strategy was generated.
Kalman filters are explicitly framed in terms of predictions, but Kalman filters don’t sit around saying “I’m a Kalman filter! I make predictions!”. They just do things which are usefully described as “making predictions” from the outside.
It is possible to build an effective temperature control system that doesn’t involve predictions at all; you can precompute a strategy (like “turn heater on below X temp, turn it off above Y temp”)
What counts as a “prediction” is in the eye of the beholder.
It’s possible to find a well documented Kalman filter and describe it’s behavior on the level of what mathematical operations are performed without ever thinking of it as “predicting” anything. “What do you mean ‘predicting’? It’s a computer, it can’t think! it’s just adding this number to that number and multiplying by these other numbers!”.
It is equally possible to give a teleological explanation of the bang bang controller. “It thinks that the temperature will follow the right trajectory iff it turns on the heater when it’s too cold” perfectly describes the behavior. The bimetallic strip closing the circuit functions as a prediction that more heat will put the temp on the right trajectory, and the control system “works” to the extent that this prediction is accurate.
It’s possible to build a temperature control system without thinking in terms of predictions, but it’s not possible to build one that cannot be usefully thought of as such. If you ever find a system that you can’t describe as modeling the process it’s controlling, it won’t work. If you show me the structure of your controller and tell me that it works well, I can use that to infer things about the process you’re using it to control (e.g. if bang bang works well, there’s negligible lag between the output and the state).
This might sound like “semantics”, but it is actually necessary in order to create good control systems. If you design your temperature controller to wait until it knows what the temperature is before choosing an output, and make “predictions” about what will happen without driving them to what you want to happen, you will have a crummy controller. Maybe crummy is “good enough”, but it will fail in any application that demands good performance. This is stuff I had to figure out before I could get the performance I wanted out of an actual temperature controller. Looking at a PID controller as if it’s “making predictions” allowed me to see where the predictions were wrong, implement better predictions by incorporating the additional information I was using to judge, and set gains that keep the expected value of temperature equal to the setpoint. The result is better control than is possible otherwise.
Okay, so let’s return to the question of saturation and connect it back to human cognition.
An optimal controller with implicit “self confidence” will maintain the prediction that its output will realize the prediction. What happens when the heating element is less powerful than the one it was programmed to expect?
The controller will keep predicting it will hit the setpoint, keep putting out enough heat that it “should” reach the setpoint, and keep being wrong.
If it has an integral gain, it will notice this and try to add more and more heat until it stops being wrong. If it can’t, it’s going to keep asking for more and more output, and keep expecting that this time it’ll get there. And because it lacks the control authority to do it, it will keep being wrong, and maybe damage its heating element by asking more than they can safely do. Sound familiar yet?
So what’s the fix? Update its self model to include this limitation, of course.
But what happens to its predictions when this happens? What happens to the temperature that it acts to realize? It drops.
It is now functionally identical to a controller which controls to an optimum trajectory, rather than controlling to the state the controls engineer wishes it were already at. You can describe it as “trying to be at 70f” if you add enough epicycles of “When it can”/”without destroying itself”, etc. Or you can describe it more simply as trying to regulate to an optimal trajectory towards 70f, without epicycles. Updating on one’s inability to achieve a goal necessarily results in no longer trying for that goal, or corrupting your idea of “trying” until it no longer pays the rent.
So what’s the fix for people?
If you find someone beating his head against a wall in attempt to get through, it’s because he’s thinking “I’m gonna get through, dammit!”. Get him to sit with the question “Are you, though? Really?”, and he will stop trying, because obviously no, lol.
If he doesn’t want to look, a good bet is that it’s because he doesn’t have any other trajectory to fall back to. Show him that he can walk around, and all of a sudden you’ll find it much easier to convince him that he can’t bash his head through the wall.
Just like the temperature controller thing, this is a real thing that produces real results. Even my post showing how I helped someone untangle his debilitating chronic pain over a few PMs is an example of this. You might not think of “Nerve damage pain” as motivated cognition, but the suffering comes from refused updates, he was refusing the updates because it would have meant that he could no longer work towards something important, and helping him see how to control without denying reality is what actually helped.
If it has an integral gain, it will notice this and try to add more and more heat until it stops being wrong. If it can’t, it’s going to keep asking for more and more output, and keep expecting that this time it’ll get there. And because it lacks the control authority to do it, it will keep being wrong, and maybe damage its heating element by asking more than they can safely do. Sound familiar yet?
From tone and context, I am guessing that you intend for this to sound like motivated reasoning, even though it doesn’t particularly remind me of motivated reasoning. (I am annoyed that you are forcing me to guess what your intended point is.)
I think the key characteristic of motivated reasoning is that you ignore some knowledge or model that you would ordinarily employ while under less pressure. If you stay up late playing Civ because you simply never had a model saying that you need a certain amount of sleep in order to feel rested, then that’s not motivated reasoning, it’s just ignorance. It only counts as motivated reasoning if you, yourself would ordinarily reason that you need a certain amount of sleep in order to feel rested, but you are temporarily suspending that ordinary reasoning because you dislike its current consequences.
(And I think this is how most people use the term.)
So, imagine a scenario where you need 100J to reach your desired temp but your heating element can only safely output 50J.
If you were to choose to intentionally output only 50J, while predicting that this would somehow reach the desired temperature (contrary to the model you regularly employ in more tractable situations), then I would consider that a central example of motivated reasoning. But your model does not seem to me to explain how this strategy arises.
Rather, you seem to be describing a reaction where you try to output 100J, meaning you are choosing an action that is actually powerful enough to accomplish your goal, but which will have undesirable side-effects. This strikes me as a different failure mode, which I might describe as “tunnel vision” or “obsession”.
I suppose if your heating element is in fact incapable of outputting 100J (even if you allow side-effects), and you are aware of this limitation, and you choose to ask for 100J anyway, while expecting this to somehow generate 100J (directly contra the knowledge we just assumed you have), then that would count as motivated reasoning. But I don’t think your analogy is capable of representing a scenario like this, because you are inferring the controller’s “expectations” purely from its actions, and this type of inference doesn’t allow you to distinguish between “the controller is unaware that its heating element can’t output 100J” from “the controller is aware, but choosing to pretend otherwise”. (At least, not without greatly complicating the example and considering controllers with incoherent strategies.)
Meta-level feedback: I feel like your very long comment has wasted a lot of my time in order to show off your mastery of your own field in ways that weren’t important to the conversation; e.g. the stuff about needing to react faster than the thermometer never went anywhere that I could see, and I think your 5-paragraph clarification that you are interpreting the controller’s actions as implied predictions could have been condensed to about 3 sentences. If your comments continue to give me similar feelings, then I will stop reading them.
I think the key characteristic of motivated reasoning is that you ignore some knowledge or model that you would ordinarily employ while under less pressure.
A pretty standard definition of motivated reasoning is that it is reasoning that is actively working towards reaching a certain preferred conclusion,
Quoting Googles AI overview (which is generally pretty terrible, but suffices here),
“Motivated reasoning is the psychological tendency to process information in a biased way, seeking out evidence that supports what we want to be true (our beliefs, desires, identity) while dismissing contradictory facts, often unconsciously, to avoid discomfort or maintain a positive self-image.”
It doesn’t require that you already have the knowledge or model, if you would otherwise acquire it if you weren’t trying to reach a certain conclusion. Failure to learn new things is far more central, because if you already have well integrated models it becomes hard to form the broken intentions in the first place.
If you were to choose to intentionally output only 50J, while predicting that this would somehow reach the desired temperature (contrary to the model you regularly employ in more tractable situations), then I would consider that a central example of motivated reasoning.
I think there are a lot of missing pieces in your picture here. How do you operationalize “intentionally”, for one? Like, how do you actually test whether a system was “intentional” or “just did a thing”? If a system can’t put out more than 50j, in what sense is 50j the intention and not 100 or “more” or something else?
Rather, you seem to be describing a reaction where you try to output 100J, meaning you are choosing an action that is actually powerful enough to accomplish your goal, but which will have undesirable side-effects.
Well, not necessarily, which is why I said “and maybe”. If I program in a maximum pulse width, the controller upstream doesn’t know about it. It puts out a new value, which maybe would or maybe wouldn’t be enough, but it can’t know. All it knows is that it didn’t work this time, and it’s not updating on the possibility that maybe failing the last twenty times in a row means the temperature won’t actually reach the setpoint.
I suppose if your heating element is in fact incapable of outputting 100J (even if you allow side-effects), and you are aware of this limitation, and you choose to ask for 100J anyway, while expecting this to somehow generate 100J (directly contra the knowledge we just assumed you have), then that would count as motivated reasoning.
That is far closer to the point. The controller makes motions that would work under its model of the world… in expectation, without any perceived guarantee of this being reality… and in reality that isn’t happening.
The problem now is in the interaction between the meta level and the object level.
On the object level, the controller is still forming its conclusions of what will happen based on what it wants to happen. This is definitionally motivated cognition in a sense, but it’s only problematic when the controller fails. The object level controller itself, by definition of “object level”, is in the business of updating reality not its model of reality. The problematic sense comes in when the meta level algorithm that oversees the object level controller chooses not to deliver all the information to the object level controller because that would cause the controller to stop trying, and the meta level algorithm doesn’t think that’s a good idea.
Let’s look at the case of the coach saying “You gotta BELIEVE!”. This is an explicit endorsement of motivated reasoning. The motivational frame he’s operating in is that you expect to win, figure out what you gotta do to get there, and then do the things. The problem with giving this object level controller full info is that “Oh, I’m not gonna win” is a conclusion it might reach, and then what actions will it output? If you’re not gonna win, what’s it matter what you do next? If full effort is costly, you’re not going to do it when you’re not going to win anyway.
When you shift from controlling towards “win” to controlling towards the things that maximize chances of winning, then “I’m not gonna win though” becomes entirely irrelevant. Not something you have to hide from the controller, just something that doesn’t affect decision making. “Okay so I’m gonna lose. I’m still going to put in 100% effort because I’m going to be the person who never loses unnecessarily”.
The motivated reasoning, and explicit endorsement of such, comes from the fact that being fully honest can cause stupid reactions, and if you don’t know how to use that additional information well, updating on it can result in stupider actions (from the perspective of the meta layer). Same thing with “No, this dress doesn’t make your ass look fat honey”/”She’s just gonna get upset. Why would I upset her?” coming from a person who doesn’t know how to orient to difficult realities.
because you are inferring the controller’s “expectations” purely from its actions, and this type of inference doesn’t allow you to distinguish between “the controller is unaware that its heating element can’t output 100J” from “the controller is aware, but choosing to pretend otherwise”.
Oh, no, you can definitely distinguish. The test is “What happens when you point at it?”. Do they happily take the correction, or do they get grumpy at you and take not-fully-effective actions to avoid updating on what you’re pointing at? Theoretically it can get tricky, but the pretense is rarely convincing, in practice.
With simple bimetallic thermostat, it’s pretty clear from inspection that there’s just no place to put this information, so it’s structurally impossible for it to be aware of anything else. Alternatively, if you dig through the code and find a line “while output>maxoutput, temp—”, you can run the debugger and watch the temperature estimate get bullshitted as necessary in order to maintain the expectation.
Meta-level feedback:
I can’t help but notice that the account you’re offering is fairly presumptuous, makes quite a few uncharitable assumptions, and doesn’t show a lot of interest in learning something like “Oh, the relevance of the response time thing wasn’t clear? I’ll try again from another angle”. It’d be a lot easier to take your feedback the way you want it taken if you tried first to make sure you weren’t just missing things that I’d be happy to explain.
If you’re wed to that framing then I agree it’s probably a waste of your time continue. If you’re interested in receiving meta level feedback yourself, I can explain how I see things, why, and we can find out together what holds up and what doesn’t.
Amusingly, this would require neither of us controlling towards “being right” and instead controlling towards the humility/honesty/meta-perspective-taking/etc that generates rightness. Might be an interesting demonstration of the thing I’m trying to convey, if you want to try that.
Also, sorry if it’s gotten long again. I’m pretty skeptical that a shorter solution exists at all, but if it does I certainly can’t find it. Heck, I’d be pleasantly surprised if it all made sense at this length.
At some point, a temperature control system needs to take actions to control the temperature. Choosing the correct action depends on responding to what the temperature actually is, not what you want it to be, or what you expect it to be after you take the (not-yet-determined) correct action.
If you are picking your action based on predictions, you need to make conditional predictions based on different actions you might take, so that you can pick the action whose conditional prediction is closer to the target. And this means your conditional predictions can’t all be “it will be the target temperature”, because that wouldn’t let you differentiate good actions from bad actions.
It is possible to build an effective temperature control system that doesn’t involve predictions at all; you can precompute a strategy (like “turn heater on below X temp, turn it off above Y temp”) and program the control system to execute that strategy without it understanding how the strategy was generated, and in that case it might not have models or make predictions at all. But if you were going to rely on predictions to pick the correct action, it would be necessary to make some (conditional) predictions that are not simply “I will succeed”.
Optimal controls is something I do professionally, and the (reasonable) misconceptions you have about controls are exactly the kind that produce the (reasonable) misconceptions get people stuck with motivated reasoning.
I’ll focus on the controls first, since it’s easier to see in simpler better defined situations first, and then tie it back in to the human failures.
So, you do have to respond to the data, obviously.
But like, the correct action also depends on what you want the temperature to be. If the jacuzzi is 100f, the correct action is different if you want it to be 101 than if you want it to be at 99.
When you actually try to build an optimal controller, ideas like “You respond to what the temperature actually is” fall apart. It takes several seconds to get a good temperature estimate from a thermometer. You read the thermometer, calculate new outputs, and change the underlying reality many times per second. By the time you’ve gathered enough data to make a decent estimate, the state changed long ago. If you’re really pushing the limits even the most recent data is out of date by the time you’ve parsed it, and this has very significant effects. This is what I spent this whole week dealing with at work, actually.
When doing optimal control you’re constantly thinking about what was your estimate in the most recent past timestep, and what it will be in the next timestep in the future. It’s rapid iteration between “I think it will be X”, “oops, lemme correct that”, “I think it will be X”. The key insight here is that this process of “oops, lemme correct that” binds future expectation to the desired value, at every future timestep.
The prediction for the next timestep will always be equal to the setpoint in an unsaturated optimal controller, exactly, because that’s what defines optimality. If you choose an output that results in 69.9f in expectation, then you could make an argument that you’re optimally controlling to 69.9f, but you’re not optimally controlling to 70f because outputting more heat would have done better by that metric.
The obvious response to this is “What if it can’t get to 70f by the next timestep, even at max power!?”, and the answer is that this would mean it’s saturated. Saturation changes things in interesting ways which I’ll return to at the end.
You don’t need to make conditional predictions at all. Most control systems don’t. A Kalman filter coupled with an optimal control law will make unconditional predictions only, for example.
It’s true that if you try to reason the way you describe and bullshit your answers you’ll get bad results, but it doesn’t have anything to do with what I’m saying. Even if I reason “If I don’t eat next month, I’ll starve, and that’s bad”, the thing that follows is “so I’m not gonna do that”. At the end of the day, I expect to not starve, because I intend to not starve.
Control systems never understand how the strategy was generated. Control systems are what do the control. They implement the strategy. Controls engineers are what do the understanding of how the strategy was generated.
Kalman filters are explicitly framed in terms of predictions, but Kalman filters don’t sit around saying “I’m a Kalman filter! I make predictions!”. They just do things which are usefully described as “making predictions” from the outside.
What counts as a “prediction” is in the eye of the beholder.
It’s possible to find a well documented Kalman filter and describe it’s behavior on the level of what mathematical operations are performed without ever thinking of it as “predicting” anything. “What do you mean ‘predicting’? It’s a computer, it can’t think! it’s just adding this number to that number and multiplying by these other numbers!”.
It is equally possible to give a teleological explanation of the bang bang controller. “It thinks that the temperature will follow the right trajectory iff it turns on the heater when it’s too cold” perfectly describes the behavior. The bimetallic strip closing the circuit functions as a prediction that more heat will put the temp on the right trajectory, and the control system “works” to the extent that this prediction is accurate.
It’s possible to build a temperature control system without thinking in terms of predictions, but it’s not possible to build one that cannot be usefully thought of as such. If you ever find a system that you can’t describe as modeling the process it’s controlling, it won’t work. If you show me the structure of your controller and tell me that it works well, I can use that to infer things about the process you’re using it to control (e.g. if bang bang works well, there’s negligible lag between the output and the state).
This might sound like “semantics”, but it is actually necessary in order to create good control systems. If you design your temperature controller to wait until it knows what the temperature is before choosing an output, and make “predictions” about what will happen without driving them to what you want to happen, you will have a crummy controller. Maybe crummy is “good enough”, but it will fail in any application that demands good performance. This is stuff I had to figure out before I could get the performance I wanted out of an actual temperature controller. Looking at a PID controller as if it’s “making predictions” allowed me to see where the predictions were wrong, implement better predictions by incorporating the additional information I was using to judge, and set gains that keep the expected value of temperature equal to the setpoint. The result is better control than is possible otherwise.
Okay, so let’s return to the question of saturation and connect it back to human cognition.
An optimal controller with implicit “self confidence” will maintain the prediction that its output will realize the prediction. What happens when the heating element is less powerful than the one it was programmed to expect?
The controller will keep predicting it will hit the setpoint, keep putting out enough heat that it “should” reach the setpoint, and keep being wrong.
If it has an integral gain, it will notice this and try to add more and more heat until it stops being wrong. If it can’t, it’s going to keep asking for more and more output, and keep expecting that this time it’ll get there. And because it lacks the control authority to do it, it will keep being wrong, and maybe damage its heating element by asking more than they can safely do. Sound familiar yet?
So what’s the fix? Update its self model to include this limitation, of course.
But what happens to its predictions when this happens? What happens to the temperature that it acts to realize? It drops.
It is now functionally identical to a controller which controls to an optimum trajectory, rather than controlling to the state the controls engineer wishes it were already at. You can describe it as “trying to be at 70f” if you add enough epicycles of “When it can”/”without destroying itself”, etc. Or you can describe it more simply as trying to regulate to an optimal trajectory towards 70f, without epicycles. Updating on one’s inability to achieve a goal necessarily results in no longer trying for that goal, or corrupting your idea of “trying” until it no longer pays the rent.
So what’s the fix for people?
If you find someone beating his head against a wall in attempt to get through, it’s because he’s thinking “I’m gonna get through, dammit!”. Get him to sit with the question “Are you, though? Really?”, and he will stop trying, because obviously no, lol.
If he doesn’t want to look, a good bet is that it’s because he doesn’t have any other trajectory to fall back to. Show him that he can walk around, and all of a sudden you’ll find it much easier to convince him that he can’t bash his head through the wall.
Just like the temperature controller thing, this is a real thing that produces real results. Even my post showing how I helped someone untangle his debilitating chronic pain over a few PMs is an example of this. You might not think of “Nerve damage pain” as motivated cognition, but the suffering comes from refused updates, he was refusing the updates because it would have meant that he could no longer work towards something important, and helping him see how to control without denying reality is what actually helped.
From tone and context, I am guessing that you intend for this to sound like motivated reasoning, even though it doesn’t particularly remind me of motivated reasoning. (I am annoyed that you are forcing me to guess what your intended point is.)
I think the key characteristic of motivated reasoning is that you ignore some knowledge or model that you would ordinarily employ while under less pressure. If you stay up late playing Civ because you simply never had a model saying that you need a certain amount of sleep in order to feel rested, then that’s not motivated reasoning, it’s just ignorance. It only counts as motivated reasoning if you, yourself would ordinarily reason that you need a certain amount of sleep in order to feel rested, but you are temporarily suspending that ordinary reasoning because you dislike its current consequences.
(And I think this is how most people use the term.)
So, imagine a scenario where you need 100J to reach your desired temp but your heating element can only safely output 50J.
If you were to choose to intentionally output only 50J, while predicting that this would somehow reach the desired temperature (contrary to the model you regularly employ in more tractable situations), then I would consider that a central example of motivated reasoning. But your model does not seem to me to explain how this strategy arises.
Rather, you seem to be describing a reaction where you try to output 100J, meaning you are choosing an action that is actually powerful enough to accomplish your goal, but which will have undesirable side-effects. This strikes me as a different failure mode, which I might describe as “tunnel vision” or “obsession”.
I suppose if your heating element is in fact incapable of outputting 100J (even if you allow side-effects), and you are aware of this limitation, and you choose to ask for 100J anyway, while expecting this to somehow generate 100J (directly contra the knowledge we just assumed you have), then that would count as motivated reasoning. But I don’t think your analogy is capable of representing a scenario like this, because you are inferring the controller’s “expectations” purely from its actions, and this type of inference doesn’t allow you to distinguish between “the controller is unaware that its heating element can’t output 100J” from “the controller is aware, but choosing to pretend otherwise”. (At least, not without greatly complicating the example and considering controllers with incoherent strategies.)
Meta-level feedback: I feel like your very long comment has wasted a lot of my time in order to show off your mastery of your own field in ways that weren’t important to the conversation; e.g. the stuff about needing to react faster than the thermometer never went anywhere that I could see, and I think your 5-paragraph clarification that you are interpreting the controller’s actions as implied predictions could have been condensed to about 3 sentences. If your comments continue to give me similar feelings, then I will stop reading them.
A pretty standard definition of motivated reasoning is that it is reasoning that is actively working towards reaching a certain preferred conclusion,
Quoting Googles AI overview (which is generally pretty terrible, but suffices here),
“Motivated reasoning is the psychological tendency to process information in a biased way, seeking out evidence that supports what we want to be true (our beliefs, desires, identity) while dismissing contradictory facts, often unconsciously, to avoid discomfort or maintain a positive self-image.”
It doesn’t require that you already have the knowledge or model, if you would otherwise acquire it if you weren’t trying to reach a certain conclusion. Failure to learn new things is far more central, because if you already have well integrated models it becomes hard to form the broken intentions in the first place.
I think there are a lot of missing pieces in your picture here. How do you operationalize “intentionally”, for one? Like, how do you actually test whether a system was “intentional” or “just did a thing”? If a system can’t put out more than 50j, in what sense is 50j the intention and not 100 or “more” or something else?
Well, not necessarily, which is why I said “and maybe”. If I program in a maximum pulse width, the controller upstream doesn’t know about it. It puts out a new value, which maybe would or maybe wouldn’t be enough, but it can’t know. All it knows is that it didn’t work this time, and it’s not updating on the possibility that maybe failing the last twenty times in a row means the temperature won’t actually reach the setpoint.
That is far closer to the point. The controller makes motions that would work under its model of the world… in expectation, without any perceived guarantee of this being reality… and in reality that isn’t happening.
The problem now is in the interaction between the meta level and the object level.
On the object level, the controller is still forming its conclusions of what will happen based on what it wants to happen. This is definitionally motivated cognition in a sense, but it’s only problematic when the controller fails. The object level controller itself, by definition of “object level”, is in the business of updating reality not its model of reality. The problematic sense comes in when the meta level algorithm that oversees the object level controller chooses not to deliver all the information to the object level controller because that would cause the controller to stop trying, and the meta level algorithm doesn’t think that’s a good idea.
Let’s look at the case of the coach saying “You gotta BELIEVE!”. This is an explicit endorsement of motivated reasoning. The motivational frame he’s operating in is that you expect to win, figure out what you gotta do to get there, and then do the things. The problem with giving this object level controller full info is that “Oh, I’m not gonna win” is a conclusion it might reach, and then what actions will it output? If you’re not gonna win, what’s it matter what you do next? If full effort is costly, you’re not going to do it when you’re not going to win anyway.
When you shift from controlling towards “win” to controlling towards the things that maximize chances of winning, then “I’m not gonna win though” becomes entirely irrelevant. Not something you have to hide from the controller, just something that doesn’t affect decision making. “Okay so I’m gonna lose. I’m still going to put in 100% effort because I’m going to be the person who never loses unnecessarily”.
The motivated reasoning, and explicit endorsement of such, comes from the fact that being fully honest can cause stupid reactions, and if you don’t know how to use that additional information well, updating on it can result in stupider actions (from the perspective of the meta layer). Same thing with “No, this dress doesn’t make your ass look fat honey”/”She’s just gonna get upset. Why would I upset her?” coming from a person who doesn’t know how to orient to difficult realities.
Oh, no, you can definitely distinguish. The test is “What happens when you point at it?”. Do they happily take the correction, or do they get grumpy at you and take not-fully-effective actions to avoid updating on what you’re pointing at? Theoretically it can get tricky, but the pretense is rarely convincing, in practice.
With simple bimetallic thermostat, it’s pretty clear from inspection that there’s just no place to put this information, so it’s structurally impossible for it to be aware of anything else. Alternatively, if you dig through the code and find a line “while output>maxoutput, temp—”, you can run the debugger and watch the temperature estimate get bullshitted as necessary in order to maintain the expectation.
I can’t help but notice that the account you’re offering is fairly presumptuous, makes quite a few uncharitable assumptions, and doesn’t show a lot of interest in learning something like “Oh, the relevance of the response time thing wasn’t clear? I’ll try again from another angle”. It’d be a lot easier to take your feedback the way you want it taken if you tried first to make sure you weren’t just missing things that I’d be happy to explain.
If you’re wed to that framing then I agree it’s probably a waste of your time continue. If you’re interested in receiving meta level feedback yourself, I can explain how I see things, why, and we can find out together what holds up and what doesn’t.
Amusingly, this would require neither of us controlling towards “being right” and instead controlling towards the humility/honesty/meta-perspective-taking/etc that generates rightness. Might be an interesting demonstration of the thing I’m trying to convey, if you want to try that.
Also, sorry if it’s gotten long again. I’m pretty skeptical that a shorter solution exists at all, but if it does I certainly can’t find it. Heck, I’d be pleasantly surprised if it all made sense at this length.
Sure, give me meta-level feedback.