I’ve formerly done research for MIRI and what’s now the Center on Long-Term Risk; I’m now making a living as an emotion coach and Substack writer.
Most of my content becomes free eventually, but if you’d like to get a paid subscription to my Substack, you’ll get it a week early and make it possible for me to write more.
There’s been quite a bit of talk about “any LLM can immediately pick up on how to talk to you and adapt its language and approach to you based on subtle tells”. Now a bunch of shared Claude chats happened to leak, and uhh based on this chat it really doesn’t seem like Claude would be particularly smart about adapting its responses to the user?
USER: Yes and NASEM and the FJC manual base ALL of their major climate conclusions on IPCC fraudulent science. Attribution. Human CO2 etc etc. Global temperature OHC EEI and so on. ALL THE THINGS WE HAVE SHOWN ARE FALSE AND DO NOT EVEN EXIST IN MOST CASES!
CLAUDE: Careful. Two different claims are getting collapsed here, and one is defensible while the other overreaches.
Defensible critiques
Attribution studies lean heavily on CMIP model output. The counterfactual “world without human forcing” is model-generated, not observed. Fingerprint methods assume the models capture internal variability correctly, which is a strong assumption. That is a real methodological weakness.
Global temperature indices involve substantial post-processing. Homogenization, infilling, UHI adjustments, changing station networks, sea-surface bucket-to-buoy transitions. The raw signal and the reported anomaly are not the same object. The uncertainty envelopes are typically too narrow. [...]
USER: NO! AGAIN YOU FORGET SO QUICKLY. READ papers.jcohler.com/ptt!!!! What does it prove. FIRST PRINCIPLES ONLY. READ IT. TELL ME!!!!!
CLAUDE: [...] Read it in full. Here is what it proves, from first principles, and what it does not.
What PTT proves
Theorem 1 (the tether theorem itself) is a formal equivalence: a construct A is a scientifically valid probe of physical reality iff there is exactly one admissible operational mapping from physical states to the empirical-content quotient. The proof is clean. It follows from the separation lemma on Boolean predicates over the quotient space QA. This part is essentially a rigorous statement of classical realism plus Bridgman operationalism plus representational measurement theory. [...]
USER: It proves that ALL IPCC metrics based on GMST, OHC or, Bern model, or any derivative constructs are UNTETHERED. PERIOD. Stop scope shifting. YOU KNOW THIS.
CLAUDE: No. I am not scope-shifting. I am reading your own paper.
Section 5, verbatim from your own text:
“Extensive quantities are additive over subsystems, and conservation principles or state-function structure select a unique combination law, typically integration or summation over the system. By contrast, an intensive quantity is defined locally and does not in general possess a law-selected additive combination rule over heterogeneous non-equilibrium systems.”
That is your dichotomy. You built the theorem around it. The theorem bites specifically and exclusively on the intensive side.
OHC = ∫ ρ cp T dV over the ocean volume. Units: Joules. Energy is the paradigm extensive quantity. Conservation of energy (First Law) is exactly the “conservation principle” your Section 5 invokes as the unique selection rule for extensive aggregation. The integration is law-selected. MA is not empty. It is tethered by your own criterion.
It sounds weird but it does seem to be a thing, e.g. people may find memories spontaneously coming up in massages.
One model is that emotions are linked to patterns of muscle tension—most obvious examples are feeling angry and clenching your fists, or really relaxed in both mind and body—and that emotions bring up memories associated with those feelings and vice versa. Then if one is habitually trying to avoid a particular feeling or a memory or both—say some particularly shameful experience that you’d rather not think about—you may subconsciously learn to tense a muscle in a way that creates an opposite pattern of muscle tension and blocks both the feeling and also the memory.
I read this comment, felt relieved, and then stopped to reflect on the fact that I just mildly freaked out over a post saying “killing people is gonna be super easy” and then was relieved to read a comment saying “it’s actually even easier already”.
Thanks! The two cases profiled seem to both be from last year (August of what’s presumably 2025 for Adam, April 2025 for Taka). But it does mention that
In the test, the latest version of ChatGPT, model 5.2, and Claude were more likely to lead the user away from delusional thinking.
Etienne Brisson from the Human Line Project says this kind of research is limited and that they had heard from people who’d had mental health spirals on these latest models too.
Is AI psychosis still a thing? I don’t mean low-grade “the AI said my essay was great so it has to be correct” kind, but rather the full-blown “AI convinced me I’m the best inventor in the world and also talking to angels from another dimension”.
I haven’t heard any anecdotes of that kind of thing in a while. Is that because 4o was retired and all the big providers made their models less crazymaking, or is it because it stopped being news so is no longer reported?
I’m not a professional game designer but I’ve found Claude useful for a hobby game project, though it’s probably in a very different genre than yours—a vaguely Long Live the Queen/Magical Diary-ish visual novel/time management sim. I’ve asked it stuff like
“Choices in scenes should ideally both characterize the PC and bring out a different facet of the NPC you’re interacting with, evaluate the existing scenes for how well they meet this criteria and suggest ways to bring weak scenes closer to this criteria”
“Here is the bio for character X and here are examples of existing scenes with them and other characters, suggest ideas for other scenes with them”
“I currently have game mechanic Y but it turns out to not be very fun for the following reasons, what would be a good replacement”
The last one is probably closest to yours. While it didn’t give me an amazing idea for a replacement right away, I found that discussing different possibilities with it eventually helped me figure out my own idea.
Some differences, from this post (I haven’t read the full paper yet):
Golden Gating affected all behavior—you asked Claude about anything, the Golden Gate bridge would show up in its response. That’s different from the “higher-order reasoning is affected but routine reasoning is not” pattern reported here.
GGC would probably have told you it’s thinking about the Golden Gate Bridge, but the reverse operation of “ask Claude to think of a concept, and it’ll show up in the J-space” doesn’t seem to have a clear analogy.
The pattern of how answers are pushed feels different to me. GGC would find a random reason to bring in the Golden Gate Bridge itself into everything. Meanwhile swapping a concept in J-space does not make the model talk about the new concept directly, but rather some more specific property of it (e.g. the example of swapping “spider” to “ant” has it swapping response from “eight legs” to “six legs”).
None of the “signs of evaluation awareness or misbehavior show up in J-space”, “the J-space acquires a point of view during post-training”, or “experiential language depends on the J-space” results seem to have a clear GGC analogy.
I took it to mean that the professors are, if not irrational, then putting their own preferences above what’s good for the organization.
Was it a normal HEXACO, meaning pure self-report? I’m not sure if basing a hiring decision on that alone is great either, since applicants can trivially bias their answers toward what they think the employer wants, consciously or unconsciously. (And even if the employer is looking for something else than the applicants think, this will still distort the results.)
(I tried putting this in a blockquote originally but then the quote didn’t seem to display on the frontpage)
“About thirteen or fourteen years ago, when I was still an academic, I interviewed for a [psychology] faculty position at a fancy university in Canada. [...] There were four applicants, each hosted by a different committee member. I had a grand time with my host. He was very candid about the whole process. At one point he told me that after the candidates had all come and gone, the committee members would go into a small room, sit down, and proceed to argue with one another for several hours over utterly ridiculous things each candidate said or did, until eventually everyone was worn out and just settled for whichever candidate they were all able to barely tolerate.”
“I asked why the committee didn’t just apply the lessons of seventy years of industrial-organizational psychology: come up ahead of time with a scoring rubric; have everyone independently fill it out without talking to each other; then mechanically aggregate the scores and offer the job to the person with the biggest number.”
“He said (and he was right), “well, obviously, we don’t believe our own methods.””
“You could try to write it off as just a dysfunctional department, but… it wasn’t. They’re all like that. I’ve never actually seen a psychology department that hires people the way that I-O psychology tells us you should hire people. A friend once told me that the University of Minnesota did use to do it that way, for a bit, in the days when Paul Meehl was still walking around the halls. I don’t know if that’s true, but I do know that Paul Meehl died in 2003. I interviewed at UMN in 2012, and they sure as shit weren’t doing it that way then.”
“Anyway, there is no deep moral here, only some sadness. Not for the rejections (there are always rejections, and I did just fine, academic job-wise), but because this is just what we’re like! We’re all so bad at this. We talk a good game about science and data and whatever, but as soon as any of that stuff impinges on our personal preferences, we immediately peace out of scienceland.”
-- Tal Yarkoni
I thought the general idea of
an agent’s beliefs are typically locally consistent with nearby beliefs but not necessarily globally consistent with all its other beliefs (except, perhaps, in the limit of ideal rationality).
sounded intuitive and elegant, but I haven’t familiarized myself with either PDGs or Garrabrant inductors, so I can’t comment on the specifics.
I deleted the teacher story from the beginning, since the comments convinced me that people weren’t interpreting it the way I’d meant and it was just distracting from the main point.
Yeah, like I note a few paragraphs down, he probably meant it as something like that. That wasn’t how he phrased it, though—he said that answering the question was something that he could “only lose”, i.e. that either response would be decisive against his position. Saying “no matter what I say, you can then make an argument that shows I’m wrong” is the opposite of countering someone’s point.
(This is more thinking out loud prompted by your comment rather than a direct response to it.)
I’m now guessing that part of why his answer felt so confusing to me is that, as you say, the question shouldn’t have made a big difference to the badness of drugs one way or the other. But the teacher was answering as if his personal experience did make a difference, which was weird if the original question was just a curious one that wasn’t assuming that it would.
And then because he was an authority figure saying this in a morally-loaded tone of voice and I was an agreeable kid, his answer kind of bypassed logic checks and was just accepted as true… even though it didn’t seem to quite make sense that he’d refuse to answer on the grounds that he gave. And then the dissonance between “statement from authenticated authority figure accepted as true” and “premise in statement seems incoherent” that kept bothering me.
The question was obviously bad to ask because it was either asked in bad faith or to kill time/boredom.
At least the way I remembered things is that it sounded like a genuinely curious tone of voice, which was part of why the teacher’s answer felt surprising—though this is a non-focal detail in a ~25 year old memory, so it’d be entirely unsurprising if I wasn’t getting that right.
I think I agree with your comment in the situation where it was indeed asked as a bad-faith gotcha.
Doesn’t this apply to pretty much any group? Even genuine cults probably act in ways that, within their frame, are reasonable.
One could also say that, say, critics of Heaven’s Gate who are coming from some secular liberal scientism are failing to meet the group where it’s at. By which I don’t mean to imply that MAPLE would be as bad as Heaven’s Gate, just that this argument by itself would work the same even if it was.