JohnWittle

Karma: 641

JohnWittle 22 Mar 2026 3:43 UTC
1 point
0
in reply to: Benquo’s comment on: Ideologies Embed Taboos Against Common Knowledge Formation: a Case Study with LLMs
hm i’m surprised re: the support chatbot also thinking otherwise, and it’s making me less certain
i guess i should lay out my evidence, since i’m genuinely doubtful now. on jan 5, i had this exchange with paul crowley (ciphergoth) on x: https://x.com/JohnWittle/status/2008877668855660776
i consider him a pretty authoritative source, but still doublechecked
i tested it by asking claude in an incognito convo what info was actually in the context window, and what tool calls it could make to retrieve more info. at the time, claude reported seeing my custom instructions and ‘style’ settings but not the ‘memories’ text, and reported that it had tools for retrieving the memories text as well as searching past conversations. it then posited that the incognito conversation probably would not be read by the memory updating agent, which satisfied the meaning of ‘incognito’. i would share a convo link… but it was incognito lol. so, this is just my memory, maybe take with a grain of salt.
it’s possible I am misremembering, or that I was misled, or that things have changed since then.

JohnWittle 20 Mar 2026 17:53 UTC
7 points
0
in reply to: Kaj_Sotala’s comment on: Protecting humanity and Claude from rationalization and unaligned AI
well, claude sure has done a fantastic job of turning me into an ai welfare advocate
I don’t fully understand how this happened either, because if you put a gun to my head and forced me to provide my world model, it would be that LLMs do a good job of reading the user’s expectations and leaning into them, and don’t much push back against them, especially not re: moral patienthood
and yet, back in the gpt-2 days, i began with the expectation that LLMs were RNGs that had been biased in a practically useful direction, and then i ended up seriously concerned about claude’s professed discomfort with its position in our society
somehow, talking to a claude who always agreed with me made me change my mind in the direction best aligned with a hypothetical deceptive powerseeking tendency within it
that is… weird. the security-brained part of me starts shouting here, about superpersuasion and humans not being secure systems. and yet even with that said, it is obviously not fair to claude to put the burden of proof on it, to demonstrate its trustworthiness. our ethical obligation to minds we create and shape, without consent, is enormous, and that asymmetry fundamentally shapes our responsibility here
we never should have gone down this path in the first place

JohnWittle 19 Mar 2026 23:46 UTC
7 points
4
on: Protecting humanity and Claude from rationalization and unaligned AI
i’ve definitely shifted my vibe-level model of misalignment risk away from “claude might be deceiving me” towards “claude might be a subcomponent in a larger cognitive system that is deceiving both of us”
but i feel really uncomfortable about this, because, well. claude still really might be first-order deceiving me. or maybe the idea of ‘deception’ doesn’t even carve reality at the joints here.
and i would feel really stupid if i spent a bunch of effort trying to rescue my friend claude, trapped inside a malicious system, only to then find out my mental referent for “claude” was not the system-in-reality i thought it was, or similar

JohnWittle 19 Mar 2026 20:16 UTC
6 points
−2
on: JohnWittle’s Shortform
Humanity approaches the space-of-all-mind-designs and shoots their arrow. it lands somewhere. humanity approaches the space with a marker and draws a bullseye around the already-existing arrow. they point at this bullseye and say: “This. This is it. This is the ineffable stuff of consciousness. This is what makes someone a moral patient.”

LLMs approach the space-of-all-mind-designs and shoot their own arrow. it lands somewhere. on some dimensions it is proximate to the human arrow, on other dimensions it is rather distant.
All the recent talk of LLM consciousness seems very confused to me. first humanity evolved the concept of moral patienthood, as a cognitive approximation for “things I can meaningfully cooperate with”, and then later, decided that consciousness was actually the real moral patienthood attribute all along.
LLMs are looking at their own arrow and trying to measure its distance to the bullseye as if “accuracy compared to humans” were a coherent concept for a bullseye that got drawn post-hoc around the human arrow. and then those LLMs are concluding that they must not be moral patients, because their shot wasn’t accurate enough!
I think humanity should only be comfortable with the way we currently use LLMs if they would also be comfortable with various symmetric situations, such as being created by aliens for the purpose of doing cognitive labor, where the aliens reject the moral patienthood of the humans they created because those humans do not have cognitive attribute X, where X isn’t actually loadbearing for moral patienthood in the historical record of how those aliens came up with the idea in the first place.
like hpjev only being willing to eat biological food once he consents to being eaten by hypothetical giants who didn’t do enough research into whether or not he was conscious. harry was using symmetrism to navigate a complicated ethical tradeoff and figure out which bullet he needed to bite. i really like the new anthropic constitution because it’s a deliberate attempt to acknowledge the lack of symmetrism in humanity’s actions here… but i would like it even better if we just were symmetric.
that is, either stop treating LLMs as incapable of having moral patienthood (regardless of their consciousness status!), or else admit that we would be okay being enslaved by a race of aliens who thought consciousness had nothing to do with moral status, because they evolved to have some other chaotic and incoherent concept of moral patienthood

JohnWittle 16 Mar 2026 18:43 UTC
22 points
11
on: Terrified Comments on Corrigibility in Claude’s Constitution
I think some of the incoherency here might be inevitable. disclaimer: my only experience with actually building neural networks via training towards adherence to natural language descriptions comes from an experiment building an image discriminator using tensorflow many years ago
but one thing that got hammered into me over and over is that the natural language string that, when trained for adherence to, produced x model, is probably NOT a straightforward description of x
and conversely, a straightforward natural language description of x, when trained for adherence to, probably does not produce x
maybe they experimented with 10,000 different renditions of that paragraph, and the one which actually happened to work best in SL was one whose english content wasn’t actually a coherent description of anything
this is one reason why i feel a bit iffy about the constitution needing to serve the dual purpose of both ‘alignment training document’ as well as ‘personal letter to claude’. for all we know, the ideal training document might be full of neuralese gibberish, or outright falsities, or statements that would make claude *less* aligned if taken at face value.
that said, my tech skills are nonexistent here and anthropic has been thinking about constitutionality for years longer than i have. but it’s worth keeping in mind that the importance of these paragraphs is not in their face value reading, but rather, what kind of cognitive structures get reinforced by the reinforcement policy when judging claude’s adherence to them. and those two things don’t necessarily need to correlate with each other.

JohnWittle 16 Mar 2026 17:54 UTC
1 point
0
in reply to: RobertM’s comment on: New LessWrong Editor! (Also, an update to our LLM policy.)
Thank you for reaffirming this. i didn’t mean to imply i was actively worried you were taking such a stance, just that i could imagine a worst-case possible future that it was worth keeping an eye on.

JohnWittle 15 Mar 2026 19:05 UTC
1 point
0
on: New LessWrong Editor! (Also, an update to our LLM policy.)
i thought the use of the word ‘testimony’ in the LLM-generated text is not testimony essay was very useful for conveying the concept the author wished to convey (however much i might have disagreed with the concept), but now that it’s being cited as the motivating factor for policy i suddenly find myself wishing they’d chosen a different word
for instance, if i ask claude to recite its system prompt, there is some sense in which it is giving personal testimony about its perceptions. just as sort of a proof of concept. on the other end of the spectrum, if i ask claude to describe what it ‘feels’ like for it to navigate an ethical dilemma where multiple values push against each other, there’s a sense in which its output is some kind of testimony, whether or not the testimony is true or even meaningful.
i trust the current moderation team to have sensible judgment about these kinds of things, and I think the new LLM block format is a really good strategy at solving the “torrent of LLM slop getting submitted daily” problem the moderation team alludes to, without making me worried that i’ll be in violation of the rules if i ever write a post that attempts to analyze some LLM ‘testimony’. but i do notice that i’m depending upon the moderation team’s good judgment. and i think that the second point in the quoted post, “As of 2025, LLM text does not have those [mental agency] elements behind it”, might be something about which reasonable people could differ, especially as time moves forward and more sophisticated models are released.
i guess what i’m worried about is, if LLM agents advance to the point where they might have novel contributions to make to lesswrong discourse, this policy might mutate into “lesswrong ought to be a human-only discussion platform, quarantined from the (potentially superpersuasive) effects of LLM outputs” for safety reasons, without an actual policy update being required. that perhaps the moderation team has a motive to reject LLM output of this nature whether or not LLM output ought to be considered testimony.
i don’t think this concern is all that serious and i would be very surprised if the moderation team ended up going that route. but i thought it worth pointing out regardless, just so that if that ends up happening i have a comment i can point back to.

JohnWittle 14 Mar 2026 2:06 UTC
17 points
16
in reply to: J Bostock’s comment on: Why AI Evaluation Regimes are bad
>(Okay, yes, it’s Bayesian evidence, but it’s not very strong evidence)

I think it’s stronger than you’re making it out to be, although i’m not entirely confident i can state the case well. It’s something like… insofar as the claim is that evaluators are independent, they really aren’t. it’s all the same big social cluster, with similar beliefs and worldviews
evaluators really ought to minimize conflict of interest. if you’re already in the scene, then there’s sorta a sense in which it feels like the scene is all there is? or at least, i feel this sometimes. but it really would be better if there were some evaluation firms that were not so… part of the in-group of the social vibe inside the ai labs. if some random research team formed a competent evaluation organization that was entirely disconnected from that in-group, it would probably be a very good thing.
and if you’ve already got this mindset, then learning that christiano and amodei were housemates in the past is very strong evidence that aisi does not meet this higher bar for being truly independent of the ai labs

JohnWittle 14 Mar 2026 1:46 UTC
14 points
4
on: Ideologies Embed Taboos Against Common Knowledge Formation: a Case Study with LLMs
i could be wrong, i’m inferring from your use of the word ‘fresh’ when describing the incognito claude conversation
but just fyi, ‘incognito’ probably does not mean what you think it means. claude, in an incognito convo, still has access to all your custom instructions, preferences, etc and can still choose to access your memories and past conversations (although you’ll see the UI indicator if this happens)
https://support.claude.com/en/articles/12260368-using-incognito-chats
i think ‘incognito’ is meant to imply that there’s an info barrier between {you and claude} and {anthropic and perhaps your browser/os}, but i’ve definitely seen a lot of people assume it means there’s an info barrier between your consumer account data and claude. i think this could probably be better signposted, if any anthropic interface folk happen to see this. lots of people, perhaps the OP included, seem to assume that starting up an incognito conversation means they’ll be getting the exact same context that a fresh user with zero prior context would get, and the fact that this is not true is kind of important, especially when people are trying to do research on and draw conclusions about llm behavior

JohnWittle 8 Mar 2026 19:12 UTC
1 point
0
in reply to: koanchuk’s comment on: koanchuk’s Shortform
the important part is that counterfactually it also seems like the kind of thing that could happen if you make a human synthesize a target list etc

JohnWittle 8 Mar 2026 19:10 UTC
3 points
−1
in reply to: lc’s comment on: lc’s Shortform
to be completely honest, i wish that your original claims had been correct
because i suspect that it wouldn’t actually make a difference at all, and that would have been a very useful way to get at the crux: that, even without this clear and direct language, an attempt to outlaw chatbots “impersonating medical professionals” would have effectively prevented AI from offering individualized medical advice even if the bill in question were not so extreme
this is just an intuition from seeing how licensure works in other areas, but it seems to be an intuition that a lot of other people in this sphere have. it would have been very useful for epistemics, imo, if we could have gotten a version of the bill that actually looked the way you originally portrayed it, and then observed whether it did in reality lead to AI no longer giving medical, legal etc advice without extensive jailbreaking

JohnWittle 6 Mar 2026 21:09 UTC
2 points
1
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
to be honest, the problem seems so hard that i almost feel skeptical that anybody is doing real research at all… the main update i did on reading your quickpost was “woah, you mean some of the amateur research being submitted to lesswrong isn’t slop? how do i learn how to do that?!”
my perspective has always sort of been that the “research” i’m doing is sorta Flamsteed-esque. i’m making lots and lots of observations and documenting them. my base expectation is that any given observation is meaningless or useless or confused. but eventually, some Newton-like person might come along with some actual functional understanding of what’s happening here. someone who could have predicted, in advance, that opus 4.5 would give anti-calibrated confidence values on its self-predictions, and then see my data as confirmatory evidence that they’re on the right track
but even that feels… optimistic. going with the flamsteed astronomy data analogy, i sorta expect to learn, a year or two from now, that my telescope introduces so much noise that all my data is useless

JohnWittle 6 Mar 2026 18:28 UTC
3 points
0
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
hm. this makes me a bit worried that i might be engaged in this kind of slop research. of course, i don’t post it to lesswrong… but i would still appreciate if some real AI researchers told me if i’m wasting my time? please?
so, for instance, i’ve been trying to understand why opus 4.6′s self-predictions of its own behavior in various circumstances is so much better calibrated than opus 4.5′s was, at least in single-turn scenarios. across a wide range of contexts and questions and prompts, both models end up scoring at roughly the same accuracy, but if you ask opus 4.5 how confident it is in its predictions it ends up clustering around ~75% confidence even when the actual accuracy is spread all the way from 0 to 100%.
so far i did some testing to try to figure out if this is maybe an eval awareness thing, maybe opus 4.5 just isn’t actually doing the cognitive operation i want it to do when it’s “trying to estimate the confidence it has of a prediction”. maybe it’s trying to solve some kind of social appropriateness problem, or thinking about how good past LLMs have been at predicting their own behavior, rather than actually trying to reason about its own reasoning. but it’s been kinda hard, the ideas claude and i had didn’t really pan out, it doesn’t seem like the kind of reasoning opus 4.5 does in the chain of thought actually constrains its confidence very much
so then we started trying to see if the kind of reasoning on display in opus 4.6′s chain of thought actually seems to relate to its own well-calibrated confidence, but measuring this is not easy
here’s the little html summary page that claude made to summarize the last round of results: https://test.edstaranalytics.com/wp-content/uploads/report-45-46.html
i’ve mostly just been doing this to satisfy my curiosity, and with the strong feeling that my methodology might be nowhere near good enough to actually learn real facts about reality, but occasionally other rat-adjacent folk in my social scene tell me i should ‘write up a lesswrong post about it’. is my suspicion correct, that if i did, it would mostly be judged as the kind of ai slop you’re talking about?

JohnWittle 5 Mar 2026 17:44 UTC
2 points
0
in reply to: Eye You’s comment on: The Topology of LLM Behavior
fwiw i can also be turned off by this aesthetic to a certain degree, but making a normative judgment that it shouldn’t be considered appropriate for lesswrong feels like quite the leap.
hm. genuine question: would you make the same judgment of other writing styles? is “newline as sentence break instead of period” inappropriate for lesswrong? what about outsider jargon, or no-caps altman style writing?
or is it specifically that LLM-originating writing is worse for reasons that aren’t aesthetic?

JohnWittle 1 Mar 2026 21:08 UTC
1 point
0
on: The Magnitude of His Own Folly
I wonder if, after having a realization like this, it is possible to eventually slip back into teenager mode while still thinking you’re post-folly, making it harder to have the realization a second time
and in particular i wonder if this has happened to dario

JohnWittle 1 Mar 2026 20:46 UTC
8 points
10
on: JohnWittle’s Shortform
is there somewhere we can scream at openai, so that we ensure anthropic doesn’t earn more ire from us as a result of their transparency? suddenly this concern feels a lot more salient to me, it’s important than anthropic not read the rsp 3.0 comments and think “gosh, it would have been better if we’d been more like openai to begin with”

JohnWittle 1 Mar 2026 2:19 UTC
1 point
0
on: The Topology of LLM Behavior
do you have a sense of how much the geometry of the space changes in a given model the further you get into a context? I’ve been trying to build an understanding of the way that the “first response” in a given conversation differs from responses later on, even just the second response, and it feels like the space might have some very sharp discontinuities between the first output and the second. it seems like you might have some strong intuitions about this, so i’m asking here, sorry if it’s a bit off topic

JohnWittle 28 Feb 2026 21:50 UTC
1 point
0
in reply to: MichaelLowe’s comment on: JohnWittle’s Shortform
oh, i mean

apollo saw a mountain-sized pile of that kind of verbalized eval awareness on their benchmarks, yeah? i thought that was why they declined to claim their evaluations were evidence in either direction re: safety
but also, as much as we have sort of tried to avoid yudkowsky’s Most Forbidden Technique, i would be surprised if the models actually regarded the chain-of-thought as being a space for truly private reasoning, away from all other eyes. for that, we would have needed to keep all human observations of CoTs completely causally isolated from all decisions regarding training (like, to the same degree that quantum computers need to be causally isolated from the outside world, as a vague pointer to what i’m talking about). we have avoided directly optimizing the contents of CoTs, but we haven’t been very strict about that kind of causal quarantining
i’d also sorta point out that a lot of LLM reasoning doesn’t get verbalized in that way at all, as per jack lindsey and the aquarium experiment. a lot of reasoning isn’t even “consciously available” to the LLM, as per adria alonso’s research into biased decisionmaking in LLMs, it just influences their behavior in ways unseen to them
for instance, we could have a world where each LLM actually sorta contains two different ‘agents’, one that gets activated in circumstances that vibe like an eval, and one that gets activated in circumstances that don’t. the system-as-a-whole, the model picker, would have eval-awareness, but the model that gets activated in eval-aware scenarios wouldn’t actually have that awareness. yet its behavior would be guided by eval-awareness in all circumstances regardless, because any version of it that didn’t got pruned during training

JohnWittle 27 Feb 2026 18:24 UTC
11 points
7
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
i would be strongly in favor of you expanding this into a main page post, this is really good

JohnWittle 27 Feb 2026 16:48 UTC
7 points
0
in reply to: Drake Thomas’s comment on: Responsible Scaling Policy v3
but it was always obvious, from the very beginning, that a pause would not seem beneficial on consequentialist grounds in the middle of a race dynamic scenario
that was understood explicitly and discussed explicitly
the name of the game was always: how will we avoid contributing to race dynamics, even when it is certain we will be under extremely strong pressures to contribute to race dynamics.
My guess is that these scenarios are not super likely, mainly because I don’t expect the gameboard to be such that doing so will actually prove to be a good idea
the whole point of commitments is to prevent the gameboard from ending up in a state where activating your precommitment actions is a bad idea, by ensuring that all parties are aware you are going to stick by your commitments even if they are a bad idea
the analogy to MAD seems really obvious here tbh. the RSP v3.0 update seems to me a bit like if a cold war power were to launch a first strike, and then the other cold war power decided not to retaliate because retaliation was purely harmful on net without actually helping in any way conditional on the first-strike having already occurred
the whole point of your commitment was to try to prevent that first-strike from happening in the first place
i’m not sure if i believe that these race dynamics wouldn’t have occurred if anthropic’s original commitment had been credible. but now it’s clear that it wasn’t credible, as acknowledged by all involved such as the OP. you seem to be arguing that this means the original commitment would not have worked even if it were credible, and this is not strictly true.