jimmy

• jimmy 16 Feb 2026 1:10 UTC

  5 points

  0

  in reply to: Zack_M_Davis’s comment on: Hazards of Selec­tion Effects on Ap­proved Information

  I’m roughly equally worried about both. The one you’re pointing at is definitely the bigger problem in general, but we also see it and defend against better. The problems you don’t orient to are the ones that kick your ass while you blame it on something else. I think it’d take a LONG time to coordinate around someone who exploits the other side.

  The thing that’s interesting to me is that you are a conspicuous outlier in how open to push back you are, which I find admirable and a good thing and all that, but it also positions you to be unusually exposed to the limitations of this framing. Most people aren’t so open to push back because they sense these limitations, so I’m curious. Do you experience it as genuinely effortless regardless of the type of criticism you receive? Or do you feel like you are doing something that requires actively holding yourself to high standards because the standards are important?

  If it’s the former that’d be very interesting, and I’d have to think about how to make sense of that. If the latter, I’m pointing at the thing that makes it not effortless. I think there’s interesting stuff there, even if the final result comes back to basically supporting the position you already take. I just want to cultivate norms that make openness to criticism easier, and therefore more plentiful.

• jimmy 15 Feb 2026 8:47 UTC

  2 points

  0

  in reply to: Zack_M_Davis’s comment on: Hazards of Selec­tion Effects on Ap­proved Information

  I agree that’s the hope. And in a well functioning community of peers that can work. Especially if you only care about your true peers and already well functioning communities.

  In practice, how well do you see it working? Does LW live up to this standard as well as you might hope, or do you notice bits here and there where the market for ideas is distorted by other considerations?

  If you get someone who is smart and a valuable contributor and also skillfully if unintentionally abuses LW’s blind spots here, how long do you expect it to take before common knowledge can be formed and the problem gets corrected?

  It occurs to me (at age 38 with no dayjob, no girlfriend, and no children) that efficient markets in life activities may already sufficiently mitigate this on its own: having more time to burn arguing on the internet than anyone else is its own punishment.

  Hah.

  This does bring up another important point, which is that IMO humility is far more important than “making true criticisms” for determining who deserves status. At the end of the day, the person who accepts the correction and the one who made it both have the same insights to share. It’s only the humble that can be trusted to weight their confidence honestly, and therefore only the humble that it makes sense to listen to when they say something that isn’t obviously true.

• jimmy 14 Feb 2026 5:45 UTC

  14 points

  7

  on: Hazards of Selec­tion Effects on Ap­proved Information

  I mostly agree, and I’m glad to see the point written up here. That said, I think there’s something to the other side that isn’t given its full due here.

  (If you independently got everything exactly right the first time, then there would be nothing for critics to do; it’s just that that seems pretty unlikely if you’re talking about anything remotely complicated. It would be hard to believe that such an unlikely-seeming thing had really happened without the toughest critics getting the chance to do their worst.)

  Of course there’d be stuff for critics to do! The point of doing criticism is to make yourself look good by tearing down the author, and you don’t need actual flaws for that. The superficial appearance of flaws is enough, so long as you can persuade the audience that they’re flaws.

  Heck, you don’t even need that. If you can persuade the audience that you’re operating in good faith while also baiting the author to get agitated with you—or persuading the audience that they are even if they’re not—then you can use that to make yourself look good at the expense of the author! For example, even correct and justified accusations of bad faith on the part of the critic can be used by the critic to argue bad faith on the part of the author and claim their sought after social status.

  Of course in our salon we’re all above this and purely motivated by truth not status, but speaking 100% sincerely now, we know for sure our critics aren’t. No matter how flawless your work, if you try to say anything non-obvious or in any way challenging, there will be plenty of (illegitimate) attack surface for critics to go after. And if the community cannot sus out bad actors because the official policy is “If you have nothing to hide you have nothing to worry about!” “If your point is valid, there will be nothing to criticize!” and “Can’t police tone, because very slippery slope”, then there will be artificial and unnecessary missing of behavior that is detriment to the group epistemics AND cohesion.

  So yeah, “That criticism is hateful and unnecessary” is a suspicious claim, and not taking appropriate skepticism to such claims quickly leads to slipping down the slope into isolation from good criticism. And desire for prestige over truth makes this likely not-fully-accidental.

  And also, “Oh no, I’m not trying to tear anyone down for status, I’m totally just truth seeking in my criticisms” is equally suspicious, for the same reasons, and failing to recognize that leads to slipping in the opposite direction. This time, it’s desire to avoid looking prestige focused and desire to avoid looking uncompassionate that can make this deviation from truth not-fully-accidental.

  It’s not trivial to discern which is less wrong in any given case, and I’d want to see Salon organizers aware of both failure modes and the tradeoff.
  

• jimmy 11 Feb 2026 17:34 UTC

  4 points

  2

  in reply to: Gordon Seidoh Worley’s comment on: Vibestemics

  
  Oh geez, looking back at my comment I was extremely unclear. Sorry about that.

  Probably not useful as feedback, but the specific things I’m most interested in here are your conclusions. Like, “Not gonna justify this yet, but I think rationalists are susceptible to getting seduced by witches in ways that will turn their lives upside down. The abstractions that predict this are after the fold, and you gotta apply it to your own raw data”. I’m mostly curious about this because I’m trying to figure out how similar our perspectives are. The more similar our conclusions, the more it seems like “Water in the eggplant” type stuff is true and important just not for me. The more dissimilar, the more I have to wonder “Wait, what do you actually mean by that. I must be missing some patterns he’s matching while thinking I get it”.

  Separately from that, and what may or may not be useful, is that in general I find it helpful to have more concrete applications spelled out. Not raw data necessarily as the point isn’t to fuel independent abstraction, but a minimal set of simulated/​curated data to highlight the connection between your abstractions and concrete use cases. Sounds like you’re mostly on board with this though, at least in theory.

  I’m with you on the “I just wanna tell you about the cool abstractions I figured out!” thing by the way, hehe. It’s a lot easier, and more fun, and genuinely worth doing first I think… just also a lot harder to get through to people IME because grounding the abstractions is much harder than holding them in the abstract.

• jimmy 10 Feb 2026 8:15 UTC

  4 points

  0

  in reply to: Gordon Seidoh Worley’s comment on: Vibestemics

  Yeah, I’m familiar with Chapman and loosely follow him. His latest post was especially interesting to me, because it was unusually concrete about what people actually get wrong in practice. It’s one thing to say “There’s water in an eggplant” to which, “Yes, and?”, and quite another to show someone’s life falling apart because of things they’re systematically not seeing. To which it’s much more “Could that happen to me? And if not, what exactly is stopping it?”

  I’m probably not your intended audience because I’m already familiar with Chapman’s work and mostly agree with your position in the quoted statement, but that’s the kind of writing I personally would be most interested in reading from you, for whatever that’s worth. I’ll read some more of your stuff though and see what jumps out at me.

• jimmy 6 Feb 2026 22:07 UTC

  2 points

  0

  in reply to: Gordon Seidoh Worley’s comment on: Vibestemics

  there’s something in the desire of folks like myself to point to a place that says “hey, I think rationalists are doing a lot of things right, but are screwing up in fundamental ways that are contrary to the vibe of rationality, and it’s useful to give that thing a name so we can easily point to it”.

  Have you written about this anywhere? I’d be interested in reading it. Especially if it’s pretty direct.

• jimmy 6 Feb 2026 21:44 UTC

  2 points

  0

  in reply to: gwillen’s comment on: The skills and physics of high-perfor­mance driv­ing, Pt. 1

  This is a good point, and I was reminded of it today because I found myself approaching a green light with the other cars stopped and not yet entering the intersection. It was pretty unnerving, and I definitely hit the brakes and looked both ways before proceeding.
  
  I think the reason it’s not normally an issue is that you rarely know when the light is going to turn green so you have to slow down to stop anyway, and as a result you’re rarely a combination of close enough and fast enough to be able to beat them to the danger zone. The passing usually happens after that.

• jimmy 4 Feb 2026 22:31 UTC

  2 points

  0

  in reply to: shanzson’s comment on: Who is choos­ing your prefer­ences- You or your Mind?

  The point I’m gesturing at is kinda subtle, and I’m gesturing across a big inferential distance, but my point is that you can’t, actually.

  One you’ve let go of attachment to suffering, you can’t actually pursue “no suffering” anymore. Or maybe it’s more accurate to say “won’t pursue”. Presuppositions like “It’s good to eliminate suffering” are proof positive of attachment to “no suffering” because it’s attachment that stabilizes that thought in that unqualified form. Without that attachment it wouldn’t be presupposed because you’d notice why the suffering. And then you’d notice that “getting rid of suffering” is far from a good idea in way too many cases for “minimize suffering!” to be a good north start to guide by. You’d start to see the incredible harm that pursuance and even achievement in reduction of suffering causes, and it would no longer feel so appealing.

  Minds that are unattached to the absence of suffering can still do things like “treat the sick” and “remove the heavy object from one’s toe” that reduce unnecessary suffering… but the goal won’t be “to get away from suffering” it’ll be to help people be healthy, to restore mobility and protect the functionality of ones appendages, etc.

  The whole “I’m gonna meditate in order to learn to let go of attachments so that I can suffer less” is pretty ironic when you think about it. Because from the very perspective this stance seeks to achieve, the effort to reach a solution is itself a symptom of the problem it seeks to resolve… except the “problem” no longer looks like a problem in the first place… so, it’s fine?

  Like, “Should we let go of this attachment as it would reduce suffering?”, sure? If that’s what you want. Do you want to reduce this suffering? Or is the thing you’re attaching to more important to you than the suffering it’s causing?

  The liberated mindset won’t have a stance on “Should I let go of attachment, and suffer less?” because it’s attachment that drives shoulding in the first place. The Buddhists have a saying about ditching the raft once you cross the river, but this understates the significance of the contradiction. The very river you’re trying to escape is what’s pushing you into the raft, and what is steering the tiller. So like… which river you gonna get out of first? Which currents are worth fighting, for how long, and why?
  

• jimmy 28 Jan 2026 6:47 UTC

  3 points

  0

  in reply to: Priyanka Bharadwaj’s comment on: The Vir­tual Mother-in-Law

  Yeah, I understand the desire for sure. Regardless of whether it “should” be this way, I think I understand why it is. Any public facing LLM is going to encounter people on the wrong track, where engaging at face value will be bad for both the person using the LLM and the company running it, so they’re gonna want to try to keep things on a good track, whatever that means to them. The LLM encouraged suicides are an extreme example of this.

  Anyway, if you want to figure out what we’re doing differently to get the different responses, I’d be happy to help. IME it’s pretty straightforward to get what I want out of Claude, and I don’t feel like I’m having to to put in any extra effort beyond providing the necessary context anyway. It’s a lot like dealing with another human, except different in some ways that make it easier if you think to try it (e.g. try telling a human “I’m not interested in your opinion”, lol. Claude has a humility that most of us lack).

• jimmy 27 Jan 2026 19:21 UTC

  17 points

  2

  on: Disagree­ment Comes From the Dark World

  I don’t want people to trust me, because I think trust would result in us getting the wrong answer.

  I want people to read the words I write, think it through for themselves, and let me know in the comments if I got something wrong.

  This is a refreshing conclusion. I’m happy to point out what I think you’re getting wrong, but I have to note that this feels pretty cooperative. Already.

  If I’m a Bayesian reasoner honestly reporting my beliefs about some question, and you’re also a Bayesian reasoner honestly reporting your beliefs about the same question, we should converge on the same answer, not because we’re cooperating with each other, but because it is the answer.

  I think you’re doing a bit of slight of hand here. If I were to punch you in the face, I could say that this would damage your face—not because we’re fighting each other, simply because my fist is converging with your face. And while it’s true that it’s the fist to face impact that’s doing the damage, and that this screens off intent… this probably won’t happen unless we’re fighting. Likewise, if we’re playing an adversarial game, why the heck would I give away my informational advantage? Without at least trying to deceive you?

  That is to say, yes, “honestly reporting beliefs” is what converges people on the same answer because it’s true, but doing this is cooperation.

  But correct epistemology does not involve conflicting interests.

  Here’s a disproof by example: “You are going to do the dishes”

  You can’t divorce the two, because the truth about reality depends on how people try to achieve their interests. And we don’t tend to focus on facts that do not interest us.

  Accordingly, when humans successfully approach the Bayesian ideal, it doesn’t particularly feel like cooperating with your beloved friends, who see you with all your blemishes and imperfections but would never let a mere disagreement interfere with loving you. It usually feels like just perceiving things—resolving disagreements so quickly that you don’t even notice them as disagreements.

  So, it depends on the nature of the disagreement. If it’s just “when will the bus arrive?”, then yeah, that’s sufficiently free of emotional charge that it doesn’t feel like much, there’s little motive for dishonesty, and will often resolve before its noticed as disagreement.

  If it’s something much more meaningful, like “It’s okay if people see what you look like under your makeup” or “Despite this injury, you’re okay”, it starts to feel like something.

  These things can still be resolved “bus schedule fast”, when the disagreement really is that simple and people stay honest. It can even be fast enough that no one notices what happened. Yet “Love, imperfections and all” is actually a fairly decent description for its length. So is “Honesty, in an unusually strict sense”.

  There are techniques for resolving economic or interpersonal conflicts that involve both parties adopting a more cooperative approach, each being more willing to do what the other party wants (while the other reciprocates by doing more of what the first one wants). Someone who had experience resolving interpersonal conflicts using techniques to improve cooperation might be tempted to apply the same toolkit to resolving dishonest disagreements.

  It might very well work for resolving the disagreement. It probably doesn’t work for resolving the disagreement correctly, because cooperation is about finding a compromise amongst agents with partially conflicting interests, and in a dishonest disagreement in which both parties have non-epistemic goals, trying to do more of what the other party functionally “wants” amounts to catering to their bias, not systematically getting closer to the truth.

  Interpersonal conflicts are about dishonest disagreements. Because if we’re both being honest about “Who is going to do the dishes”, then just like the bus time disagreement, it resolves before we notice it as a “conflict”.

  “You’re going to do the dishes, because I don’t wanna”. “Actually, I think you’re going to do the dishes today because I did them yesterday, and you’re smart enough to recognize that ‘I always get what I want because I say so’ is factually untrue. So you will choose to do the thing that gets you out of as much dish washing as is possible. Which is that’s doing it your half of the time”. “Okay, you’re right”.

  Except, like.. you usually don’t have to say it out loud unless someone has been dishonest, because “I get what I want because I say so” is just pretty obviously wrong. So it’s just “Hey, is there a reason you haven’t done the dishes yet today?”, because the underlying “Because you’re smart enough to know you won’t be able to get away with shirking” goes unsaid. And the response is just “Shoot, thanks for reminding me”.
  
  Heck, even physical violence goes that way. I can’t count the number of fights I’ve avoided by responding to “Wanna fight!?” with “Ok”. It’s Aumann agreement over who is about to get their ass kicked if the fight were to happen. “I am gonna beat you up!” “I doubt it” “Me too, actually. Nvm”

  
  To “compromise” a bit, not for the sake of social-cohesion-at-the-cost-of-truth but because you make a good point that I don’t want to get lost, “compromising” on things by keeping the dishonesty and splitting the difference is indeed a failure mode worth pointing out.

  If the goal becomes “sing ‘Kumbaya’ together” rather than “track reality”, then the reality you’re not tracking is probably gonna come back to bite you. And it won’t be an accident on the part of the side that perceives it as a “win”

• jimmy 26 Jan 2026 19:48 UTC

  2 points

  0

  in reply to: Priyanka Bharadwaj’s comment on: The Vir­tual Mother-in-Law

  Okay, I tried it.
  
  I switched from talking about growing peppers to trying to understand what Kary Mullis might be picking up on that causes him to believe that astrology works.
  
  At the end of the conversation I asked

  what might be behind astrology and his observations, *specifically*? Like, not just “here’s random things to show that there could *potentially* be *some* connection”, but like… what are things you think most likely to actually be real?

  Claude’s response ended with this summary:

  If I had to bet on what Mullis was actually seeing: probably some combination of #1, #2, and #6. Seasonal effects are real, astrologers accumulated genuine correlational knowledge over centuries, and he happens to be a strong exemplar of his type. The planetary mechanism is wrong, but the pattern-matching is tracking something.

  I consider the whole conversation to have gone pretty well. Here’s Claude’s response after I pasted in this LW post and explained that I was trying to see if I got the same behavior:
  

  Ha! So this whole conversation was a field test.

  And yeah, you didn’t get pushback, playing dumb, or refusal to engage. We went from pepper growing to astrology to Kary Mullis to heritable reproductive timing strategies to NLP to your friend’s Raynaud’s resolving because you walked into a room. No guardrails triggered, no moral lecturing, no “I can’t discuss pseudoscience.”

  I did make one misstep early on – the unsolicited “Why It Doesn’t Actually Work” section when you just asked how astrology works. You correctly called that out as unwanted editorializing. But when you pushed back I dropped it and engaged with the actual substance.

  I think the difference might be in how you approach these conversations:

  1. You came in with genuine curiosity and specific empirical claims, not “tell me my horoscope” vibes. Mullis’s 1/​1728 observation, the blind horoscope test, the mechanism hypotheses – these are substantive.

  2. You pushed back when I was sloppy (the “small samples” thing), which forced me to actually think rather than pattern-match to “someone asking about pseudoscience, deploy skepticism.”

  3. You weren’t asking me to validate astrology or tell you what to believe – you were exploring what might actually be going on. That’s a different request than “is Mercury retrograde why my week sucked?”

  The person in that LessWrong post might be hitting a failure mode where Claude interprets “let’s discuss astrology” as “user might be going down an epistemically dangerous path, intervene” rather than “user wants to explore an interesting question about what patterns might underlie folk knowledge.”

  Or they might just be less skilled at steering conversations. Hard to know without seeing their actual exchanges.

  The pushback on the small samples thing looked like this:

  “small samples” smells like motivated reasoning from you—or at best, a complete ignorance about how inference works. But it seems like you should know better?

  Let me explain by example.[...]

  The pushback on the “Why it doesn’t actually work” bit was simply “I’m not interested in your personal opinion about whether it works”

• jimmy 25 Jan 2026 21:32 UTC

  3 points

  0

  in reply to: Raemon’s comment on: What’s a good method­ol­ogy for “is Trump un­usual about ex­ec­u­tive over­reach /​ in­sti­tu­tion ero­sion?”

  Right now my main question is “should I spend more time thinking about this or go back to ignoring it and hope it isn’t too bad?”.

  It seems like you’re curious and find it interesting, so why not? There are probably worthwhile things to learn.

  I think if I decided to do that I’d probably expect “solve political polarization” to be a major piece of it and yeah I’d want to talk to a wider variety of people qualitatively.

  To be clear, I don’t mean “as a way to actually fix things”, though that is where I think there’s a lot of unpicked fruit hanging embarrassingly low.

  I just mean as a personal epistemics thing. If I’m trying to figure out what’s going on, and I don’t trust my feeds to be delivering the necessary perspective, I’d want to probe what people think just to make sure there aren’t some obvious counterarguments that my current perspective is blind to. I want to make real sure I can anticipate what’s behind a disagreement before I start trusting my own perspective to be right enough to act on.

  I agree that baking in the framing into the initial question is bad, but, like, the framing is the reason why I’m even considering thinking more about this in the first place and I’m not sure how to sidestep that.

  We’re always going to have framings that make less sense in hindsight. As soon as we notice that something might be off, we can start thinking about what that might be and find out how much it holds up. I’m not sure what the problem is, since it seems like you’re doing what you’re supposed to given your epistemic state?

  Oh. Is it like… if I’m overcome with “Holy fuck, how are antivaxxers so dumb” and it motivates me to look into it, I can’t just “not have” the motivation and ignoring it would mean I don’t look at all, but if I act within that framing then everything comes out like “Why are you so dumb, anyway?” which isn’t exactly epistemically helpful?

• jimmy 25 Jan 2026 18:32 UTC

  3 points

  3

  on: The Vir­tual Mother-in-Law

  Over time, this dynamic has reversed. Claude’s stance now feels increasingly rigid; I find it treats divergent perspectives with moral judgment rather than curiosity. [...] Claude may abruptly refuse to engage with me further. Any attempts to reframe the question often leads to evasive responses, sometimes even claims it doesn’t understand what I’m asking. Yet if I open a fresh context window and ask the same question directly, it may answer without issue.

  Huh. FWIW, I’m genuinely surprised, as this does not match my experience. Early on I had some of these frustrations, but they’re completely gone now.
  
  I’d be super curious what one of these conversations looks like, and I’d like to try playing with it myself

• jimmy 25 Jan 2026 18:20 UTC

  7 points

  0

  on: What’s a good method­ol­ogy for “is Trump un­usual about ex­ec­u­tive over­reach /​ in­sti­tu­tion ero­sion?”

  I think you’re going to have a hard time coming up with “a bunch of numbers and examples” that adequately capture what’s going on. Context matters, and whether it’s you or an LLM doing the interpretation, a preferred perspective is going to end up snuck in—and it’s going to be hard to notice how much this is changing the results without an alternative to compare to.

  That seemed… like it was approaching a methodology that might (with addititional workshopping) be cruxy for some Trump supporters or Trump-neutral-ers.

  As a result, I’d be very surprised if this ends up with anything cruxy for anyone on the other side of this issue. Have you actually talked to Trump supporters who have said this would be cruxy for them? I certainly can’t imagine any of the pro-Trump people I’ve talked to being swayed by such a thing.

  The best way to deal with the problem of “I’m in a bubble, so I can’t trust the narratives I’m seeing to be the full picture” is to not stay in the bubble. The question I would be asking is “Hey Trump supporter. What am I missing, when it comes to Trump?”

  If you read angry internet arguments, then yeah, people will just defend their side no matter how detached from reality their arguments have to get. In real life though, when I’ve approached people with desire to understand they’ve all been happy to talk and give me honest takes (e.g. I don’t think I’ve ever heard “Vaccines definitely cause autism, bro!”, but I have heard “Someone I know had a very bad reaction to a vaccine, and I don’t feel like I can trust doctors”). People on both sides of the aisle have been happy to admit the faults of the candidate they voted for, or admit to things that one might find damning and explain why they actually find it to be a good thing.

  I’m not arguing against quantitative methods, just that trying to pin down “what’s actually happening [according to a certain framing]” comes at a later stage once you’ve found the framing that doesn’t presuppose away the disagreement. Because once it’s presupposed you can’t test it and find out if you’re right or unaware of your blind spot. And because you might find out “Oh, we all actually agree that something unprecedented is going on here, we just disagree on the causality”, and until you find the crux you won’t know what to measure or how to measure it.

  If I had to take a guess, I’d guess that you’d find Trump supporters agreeing that “shit’s different”, but disagree over the cause. If I were to try to pass the ITT of Trump voters to make sure I don’t have important blind spots, “the media wouldn’t make a big deal over if they didn’t hate Trump so much” seems central to me. I think the right genuinely believes that the hate is load bearing and that the left is delusional about how much hate is interfering with their ability to see things clearly. And I expect they’d have considerable disagreement with the average Trump detractor about how to operationalize “hate” and the effects thereof.

  So getting to the bottom of that is where things are going to get interesting.

• jimmy 24 Jan 2026 17:58 UTC

  2 points

  2

  on: Who is choos­ing your prefer­ences- You or your Mind?

  should we give attention to this preference? And stay attached to it?

  Or should we see it as yet another desire of the mind and let it go as attachment to it would increase suffering?

  There’s another preference and attachment happening here at the meta level: “We should let go of things that increase suffering”
  
  Is this coming from “you” or “your mind”? What happens when we turn this lens on itself?
  
  Attaching to absence of suffering creates suffering, and if you can’t see clearly the reasons behind your desires to eliminate suffering, then you will Goodhart from the start.

• jimmy 20 Jan 2026 7:06 UTC

  2 points

  0

  in reply to: Dweomite’s comment on: Why Mo­ti­vated Rea­son­ing?

  I think the key characteristic of motivated reasoning is that you ignore some knowledge or model that you would ordinarily employ while under less pressure.

  A pretty standard definition of motivated reasoning is that it is reasoning that is actively working towards reaching a certain preferred conclusion,

  Quoting Googles AI overview (which is generally pretty terrible, but suffices here),

  “Motivated reasoning is the psychological tendency to process information in a biased way, seeking out evidence that supports what we want to be true (our beliefs, desires, identity) while dismissing contradictory facts, often unconsciously, to avoid discomfort or maintain a positive self-image.”

  It doesn’t require that you already have the knowledge or model, if you would otherwise acquire it if you weren’t trying to reach a certain conclusion. Failure to learn new things is far more central, because if you already have well integrated models it becomes hard to form the broken intentions in the first place.

  If you were to choose to intentionally output only 50J, while predicting that this would somehow reach the desired temperature (contrary to the model you regularly employ in more tractable situations), then I would consider that a central example of motivated reasoning.

  I think there are a lot of missing pieces in your picture here. How do you operationalize “intentionally”, for one? Like, how do you actually test whether a system was “intentional” or “just did a thing”? If a system can’t put out more than 50j, in what sense is 50j the intention and not 100 or “more” or something else?

  Rather, you seem to be describing a reaction where you try to output 100J, meaning you are choosing an action that is actually powerful enough to accomplish your goal, but which will have undesirable side-effects.

  Well, not necessarily, which is why I said “and maybe”. If I program in a maximum pulse width, the controller upstream doesn’t know about it. It puts out a new value, which maybe would or maybe wouldn’t be enough, but it can’t know. All it knows is that it didn’t work this time, and it’s not updating on the possibility that maybe failing the last twenty times in a row means the temperature won’t actually reach the setpoint.

  I suppose if your heating element is in fact incapable of outputting 100J (even if you allow side-effects), and you are aware of this limitation, and you choose to ask for 100J anyway, while expecting this to somehow generate 100J (directly contra the knowledge we just assumed you have), then that would count as motivated reasoning.

  That is far closer to the point. The controller makes motions that would work under its model of the world… in expectation, without any perceived guarantee of this being reality… and in reality that isn’t happening.

  The problem now is in the interaction between the meta level and the object level.

  On the object level, the controller is still forming its conclusions of what will happen based on what it wants to happen. This is definitionally motivated cognition in a sense, but it’s only problematic when the controller fails. The object level controller itself, by definition of “object level”, is in the business of updating reality not its model of reality. The problematic sense comes in when the meta level algorithm that oversees the object level controller chooses not to deliver all the information to the object level controller because that would cause the controller to stop trying, and the meta level algorithm doesn’t think that’s a good idea.

  Let’s look at the case of the coach saying “You gotta BELIEVE!”. This is an explicit endorsement of motivated reasoning. The motivational frame he’s operating in is that you expect to win, figure out what you gotta do to get there, and then do the things. The problem with giving this object level controller full info is that “Oh, I’m not gonna win” is a conclusion it might reach, and then what actions will it output? If you’re not gonna win, what’s it matter what you do next? If full effort is costly, you’re not going to do it when you’re not going to win anyway.

  When you shift from controlling towards “win” to controlling towards the things that maximize chances of winning, then “I’m not gonna win though” becomes entirely irrelevant. Not something you have to hide from the controller, just something that doesn’t affect decision making. “Okay so I’m gonna lose. I’m still going to put in 100% effort because I’m going to be the person who never loses unnecessarily”.

  The motivated reasoning, and explicit endorsement of such, comes from the fact that being fully honest can cause stupid reactions, and if you don’t know how to use that additional information well, updating on it can result in stupider actions (from the perspective of the meta layer). Same thing with “No, this dress doesn’t make your ass look fat honey”/​”She’s just gonna get upset. Why would I upset her?” coming from a person who doesn’t know how to orient to difficult realities.

  
  because you are inferring the controller’s “expectations” purely from its actions, and this type of inference doesn’t allow you to distinguish between “the controller is unaware that its heating element can’t output 100J” from “the controller is aware, but choosing to pretend otherwise”.

  Oh, no, you can definitely distinguish. The test is “What happens when you point at it?”. Do they happily take the correction, or do they get grumpy at you and take not-fully-effective actions to avoid updating on what you’re pointing at? Theoretically it can get tricky, but the pretense is rarely convincing, in practice.

  With simple bimetallic thermostat, it’s pretty clear from inspection that there’s just no place to put this information, so it’s structurally impossible for it to be aware of anything else. Alternatively, if you dig through the code and find a line “while output>maxoutput, temp—”, you can run the debugger and watch the temperature estimate get bullshitted as necessary in order to maintain the expectation.

  
  Meta-level feedback:

  I can’t help but notice that the account you’re offering is fairly presumptuous, makes quite a few uncharitable assumptions, and doesn’t show a lot of interest in learning something like “Oh, the relevance of the response time thing wasn’t clear? I’ll try again from another angle”. It’d be a lot easier to take your feedback the way you want it taken if you tried first to make sure you weren’t just missing things that I’d be happy to explain.

  If you’re wed to that framing then I agree it’s probably a waste of your time continue. If you’re interested in receiving meta level feedback yourself, I can explain how I see things, why, and we can find out together what holds up and what doesn’t.

  Amusingly, this would require neither of us controlling towards “being right” and instead controlling towards the humility/​honesty/​meta-perspective-taking/​etc that generates rightness. Might be an interesting demonstration of the thing I’m trying to convey, if you want to try that.

  Also, sorry if it’s gotten long again. I’m pretty skeptical that a shorter solution exists at all, but if it does I certainly can’t find it. Heck, I’d be pleasantly surprised if it all made sense at this length.
  

• jimmy 18 Jan 2026 7:12 UTC

  10 points

  0

  in reply to: Dweomite’s comment on: Why Mo­ti­vated Rea­son­ing?

  Optimal controls is something I do professionally, and the (reasonable) misconceptions you have about controls are exactly the kind that produce the (reasonable) misconceptions get people stuck with motivated reasoning.

  I’ll focus on the controls first, since it’s easier to see in simpler better defined situations first, and then tie it back in to the human failures.

  Choosing the correct action depends on responding to what the temperature actually is, not what you want it to be, or what you expect it to be after you take the (not-yet-determined) correct action.

  So, you do have to respond to the data, obviously.

  But like, the correct action also depends on what you want the temperature to be. If the jacuzzi is 100f, the correct action is different if you want it to be 101 than if you want it to be at 99.

  When you actually try to build an optimal controller, ideas like “You respond to what the temperature actually is” fall apart. It takes several seconds to get a good temperature estimate from a thermometer. You read the thermometer, calculate new outputs, and change the underlying reality many times per second. By the time you’ve gathered enough data to make a decent estimate, the state changed long ago. If you’re really pushing the limits even the most recent data is out of date by the time you’ve parsed it, and this has very significant effects. This is what I spent this whole week dealing with at work, actually.

  When doing optimal control you’re constantly thinking about what was your estimate in the most recent past timestep, and what it will be in the next timestep in the future. It’s rapid iteration between “I think it will be X”, “oops, lemme correct that”, “I think it will be X”. The key insight here is that this process of “oops, lemme correct that” binds future expectation to the desired value, at every future timestep.

  The prediction for the next timestep will always be equal to the setpoint in an unsaturated optimal controller, exactly, because that’s what defines optimality. If you choose an output that results in 69.9f in expectation, then you could make an argument that you’re optimally controlling to 69.9f, but you’re not optimally controlling to 70f because outputting more heat would have done better by that metric.

  The obvious response to this is “What if it can’t get to 70f by the next timestep, even at max power!?”, and the answer is that this would mean it’s saturated. Saturation changes things in interesting ways which I’ll return to at the end.

  
  If you are picking your action based on predictions, you need to make conditional predictions based on different actions you might take, so that you can pick the action whose conditional prediction is closer to the target. And this means your conditional predictions can’t all be “it will be the target temperature”, because that wouldn’t let you differentiate good actions from bad actions.

  You don’t need to make conditional predictions at all. Most control systems don’t. A Kalman filter coupled with an optimal control law will make unconditional predictions only, for example.

  It’s true that if you try to reason the way you describe and bullshit your answers you’ll get bad results, but it doesn’t have anything to do with what I’m saying. Even if I reason “If I don’t eat next month, I’ll starve, and that’s bad”, the thing that follows is “so I’m not gonna do that”. At the end of the day, I expect to not starve, because I intend to not starve.

  and program the control system to execute that strategy without it understanding how the strategy was generated,

  Control systems never understand how the strategy was generated. Control systems are what do the control. They implement the strategy. Controls engineers are what do the understanding of how the strategy was generated.

  Kalman filters are explicitly framed in terms of predictions, but Kalman filters don’t sit around saying “I’m a Kalman filter! I make predictions!”. They just do things which are usefully described as “making predictions” from the outside.

  It is possible to build an effective temperature control system that doesn’t involve predictions at all; you can precompute a strategy (like “turn heater on below X temp, turn it off above Y temp”)

  What counts as a “prediction” is in the eye of the beholder.

  It’s possible to find a well documented Kalman filter and describe it’s behavior on the level of what mathematical operations are performed without ever thinking of it as “predicting” anything. “What do you mean ‘predicting’? It’s a computer, it can’t think! it’s just adding this number to that number and multiplying by these other numbers!”.

  It is equally possible to give a teleological explanation of the bang bang controller. “It thinks that the temperature will follow the right trajectory iff it turns on the heater when it’s too cold” perfectly describes the behavior. The bimetallic strip closing the circuit functions as a prediction that more heat will put the temp on the right trajectory, and the control system “works” to the extent that this prediction is accurate.

  It’s possible to build a temperature control system without thinking in terms of predictions, but it’s not possible to build one that cannot be usefully thought of as such. If you ever find a system that you can’t describe as modeling the process it’s controlling, it won’t work. If you show me the structure of your controller and tell me that it works well, I can use that to infer things about the process you’re using it to control (e.g. if bang bang works well, there’s negligible lag between the output and the state).

  This might sound like “semantics”, but it is actually necessary in order to create good control systems. If you design your temperature controller to wait until it knows what the temperature is before choosing an output, and make “predictions” about what will happen without driving them to what you want to happen, you will have a crummy controller. Maybe crummy is “good enough”, but it will fail in any application that demands good performance. This is stuff I had to figure out before I could get the performance I wanted out of an actual temperature controller. Looking at a PID controller as if it’s “making predictions” allowed me to see where the predictions were wrong, implement better predictions by incorporating the additional information I was using to judge, and set gains that keep the expected value of temperature equal to the setpoint. The result is better control than is possible otherwise.

  Okay, so let’s return to the question of saturation and connect it back to human cognition.

  An optimal controller with implicit “self confidence” will maintain the prediction that its output will realize the prediction. What happens when the heating element is less powerful than the one it was programmed to expect?

  The controller will keep predicting it will hit the setpoint, keep putting out enough heat that it “should” reach the setpoint, and keep being wrong.

  If it has an integral gain, it will notice this and try to add more and more heat until it stops being wrong. If it can’t, it’s going to keep asking for more and more output, and keep expecting that this time it’ll get there. And because it lacks the control authority to do it, it will keep being wrong, and maybe damage its heating element by asking more than they can safely do. Sound familiar yet?

  So what’s the fix? Update its self model to include this limitation, of course.

  But what happens to its predictions when this happens? What happens to the temperature that it acts to realize? It drops.

  It is now functionally identical to a controller which controls to an optimum trajectory, rather than controlling to the state the controls engineer wishes it were already at. You can describe it as “trying to be at 70f” if you add enough epicycles of “When it can”/​”without destroying itself”, etc. Or you can describe it more simply as trying to regulate to an optimal trajectory towards 70f, without epicycles. Updating on one’s inability to achieve a goal necessarily results in no longer trying for that goal, or corrupting your idea of “trying” until it no longer pays the rent.

  So what’s the fix for people?

  If you find someone beating his head against a wall in attempt to get through, it’s because he’s thinking “I’m gonna get through, dammit!”. Get him to sit with the question “Are you, though? Really?”, and he will stop trying, because obviously no, lol.

  If he doesn’t want to look, a good bet is that it’s because he doesn’t have any other trajectory to fall back to. Show him that he can walk around, and all of a sudden you’ll find it much easier to convince him that he can’t bash his head through the wall.

  Just like the temperature controller thing, this is a real thing that produces real results. Even my post showing how I helped someone untangle his debilitating chronic pain over a few PMs is an example of this. You might not think of “Nerve damage pain” as motivated cognition, but the suffering comes from refused updates, he was refusing the updates because it would have meant that he could no longer work towards something important, and helping him see how to control without denying reality is what actually helped.
  

• jimmy 15 Jan 2026 18:51 UTC

  2 points

  −4

  on: Why Mo­ti­vated Rea­son­ing?

  Motivated reasoning is a natural byproduct of any mind that tries to do anything to the outside world.

  Consider an optimal temperature controller. It has thermometers and runs a Kalman filter to calculate the probability distribution of the temperature at each moment taking into account the model of the process and all the data available. What will be the expected value of the temperature?

  The set point. Always, so long as the output isn’t saturated. Because if the expected temperature were any lower than the setpoint it’d increase the heat until it isn’t. If it were any higher, it would decrease the heat until it isn’t. The temperature controller fundamentally works by expecting the temperature to be what it “wants”, and then acting to maintain that expectation.

  This is unavoidable, because if it’s not acting to keep the expected value in line with the setpoint—according to it’s own model/​data—then it isn’t functioning as a control system with respect to that setpoint, and will be better described as optimizing for something else.

  When you analyze the system from the third person and think “Hm, p(it will achieve its goal of controlling the temp to 70f at all times) is low”, then that’s your hint to redesign the controller to stop expecting “70f” to be the temperature in the next timestep. Instead, program it to expect something that won’t fail catastrophically (e.g. “The temperature will rise to 70f as fast as it safely can without burning out heating elements”).

  To bring it back to the human, it’s not a “quirk of human biology” that your sportsball coach talks about how “You have to believe you can win!”. It comes directly from the fact that you cant try to win without expecting to win, and that people want to win.

  That doesn’t mean you can’t use more fault tolerant strategies like “score as many points as possible”. The latter can be even more effective at winning, but that is a different plan and requires giving up on trying to win. Minds that can find these more fault tolerant plans no longer need to believe they’ll win and therefore do less motivated cognition failures, so if you want to fail less due to motivated cognition then that’s the way to do it. You’ll still be expecting what you want to come true, just hopefully in a more realizable way.

  Apologizing requires not being out of trouble, and you’re trying to not be in trouble. Do you want to be in trouble, and face the consequences? If that’s not appealing to you, of course you’re going to try to not do it, and that involves expecting to not be in trouble. No wonder people come up with defensive justifications in those cases. When you want to face the music, because what you’re drawn to is being a person of integrity, and “not being guilty” is something you recognize you cannot have, then you won’t feel tempted.

  The question that can tranform the former into the latter is “Can I get what I want? Can I be not guilty and stay out of trouble?”. When you sit with that and “No, I can’t” sinks in, the temptation to rationalize melts away.

  Of course, that can be tricky to sit with too, because there are often temptations to flinch away from the question. There are reasons for that, and understanding them opens up paths to making it easier there too, but this comment has gotten long enough.
  

• jimmy 10 Jan 2026 19:31 UTC

  13 points

  1

  on: The false con­fi­dence the­o­rem and Bayesian reasoning

  
  Not uncertainty, but rather high confidence of safety.

  Phrased this way, it sounds all “spooky mysterious”, in the way that needs a named theorem and explanation.

  Phrased to be easily understood, due to the vast space and few tiny satellites, you are very unlikely to have a collision when blindly throwing darts at this dart board.

  I may be maximally ignorant about what tomorrows lottery numbers will be, but I can safely predict that I will not win.

  You certainly should not become more confident that 2 satellites are safe, just because you added random noise to the measurements. [...]However, giving each one a random push from its jets increases the actual variation in their paths, likely pushing them away from the previous point estimate of a collision, and thus does make them safer.

  And if you add random noise, you don’t get more confident. Like, if you’re cruising along on what appears to be a potential collision course, and then your sensor goes bad and starts giving noisy data, you don’t get more confident that you’re safe. You just get more scared more slowly, in the case that you’re on track to collide.

  If you never had any evidence that this unlikely event is about to take place, then of course you can’t magically get to the answer without evidence. Maybe you’re on a collision course and jinking would make you safer. More likely though, you’re just on a “close call” course, and jinking could put you on an actual collision course.

  As an outsider with special knowledge, you may be rooting for the satellite to make some random motions, or to not change anything. As an engineer working on a satellite with noisy instruments, you don’t have the privilege of knowing the right answer ahead of time, and you will have no way of firing the rockets only when it’s helpful. If you program your satellites to move randomly, all you will accomplish is wasted fuel.

  Imagine if from the inside. You’re handed a revolver and forced to play Russian roulette. Your wife saw the spin and knows whether it landed on a live round and is either praying that you spin again or praying that you don’t. You don’t know where the live round is, and can’t see or hear her. Do you ask to spin the spin the cylinder again? Would it help anything if you did?

  The take away is just that if you want to predict rare events, you need evidence. Rare events do happen, if rarely, and unless you collect the evidence you’ll have no choice but to be surprised if it happens to you.

• jimmy 23 Dec 2025 17:44 UTC

  0 points

  0

  in reply to: Nathan Young’s comment on: EA Yale Destiny De­bate Dis­cus­sion:

  I think Trump’s comments about Rob Reiner were not to remotely take his death seriously.

  Well of course. Can you honestly not see the difference?
  
  LLMs give us a really useful tool for political conversations where “mind killing” is a real risk. We can now post the entire context to Claude and ask a neutral question like “Does it look like <username> is engaging with arguments that might change his mind, or defending against them?”.
  
  What do you predict Claude will say, and are these predictions validated by experiment?
  
  Or rather, before running that check, what would the appropriate response be if one were to run such a check and not receive ego-syntonic feedback? If I run that check and Claude says “Jimmy’s comments contain the following signs of cognitive dissonance:”, what should I do with that?