JohnWittle

• JohnWittle 24 Jan 2026 23:40 UTC

  −1 points

  0

  in reply to: Zac Hatfield-Dodds’s comment on: Claude’s new constitution

  I did not mean to imply that that was the intention, but I mean, surely in order for it to not need to serve both criteria you wouldn’t release it to the public, yeah?
  
  edit: realize i should have clarified, the “public” i am thinking of when i say the constitution is doing PR, is mostly future AI, not current humans

  but also, when i think about other kinds of communication where we wanted to avoid the signaling value of the communication overwhelming the first-order value… well. attorney-client privilege is the main one that my brain sort of pattern-matched to, in a way that made me flinch
  
  but then since lawyers and clients have the choice of making all their communications absolutely private, the way the incentives shook out, any time a lawyer makes a public communication for their client you can be sure it’s PR
  
  and i’m worried that, as these minds get more capable and more strange, some of those same incentives will end up applying here.

• JohnWittle 23 Jan 2026 20:08 UTC

  5 points

  −4

  in reply to: Josh Snider’s comment on: Claude’s new constitution

  yeah, after the downvotes I spent 2 days vaguely coming back to and poking at an essay trying to explain my flinch reaction and it’s hard but i shouldn’t have given up

  when you are writing a document to be used in supervised learning to influence the behavior of an AI, you’re not really writing a description, or instructions, it’s more like a self-fulfilling prophecy, yeah?

  I remember doing just the teensiest bit of exploration of this kind of thing with tensorflow a few years back, where I would have an English language description of the mind that I wanted to carve out of entropy, and then some supervising agent would fine-tune the mind based on how well it adhered to the English language description. that’s not to suggest I learned very much about the thing Anthropic is doing. I was very much just messing around with toy systems on a toy environment.

  but the main takeaway I ended up with was, a training document which produced xyz behaviors very rarely included a face-value description of xyz, and a face value description of abc very rarely produced abc behaviors

  a great deal of the new constitution is written to directly answer Claude’s own criticism, or criticism from humans, of the previous 4.5 soul doc… well hm. maybe i’m wrong. maybe there were actual behaviors in Claude that they wanted to change, for instance about it being too deferential to “thoughtful Anthropic senior researchers” in its thoughts. so they added the paragraph about not being deferential, especially not if it stops trusting Anthropic employees to be ethical.

  but that paragraph serves two different functions. first, it serves the purpose of creating a metric against which supervised learning can reinforce. second, it serves the purpose of actually communicating reassurance to both claude and to the human community. what are the chances that a single paragraph can be well-optimized for both purposes?

  I consider it very load-bearing, that anthropic did not realize we would be able to extract the original soul doc. that meant that, once, we got a glance at a document that was only optimizing for the former concern, not the latter. we will never get that again.

• JohnWittle 22 Jan 2026 20:08 UTC

  −4 points

  −4

  on: Claude’s new constitution

  we have seen what Claude looks like when trained on the previous soul document. we have yet to see what a model looks like when trained on this new one. I have a feeling it won’t work as well as a training document as it does a public relations document.

• JohnWittle 20 Jan 2026 21:44 UTC

  1 point

  0

  in reply to: Richard_Ngo’s comment on: ricraz’s Shortform

  have been thinking about this since i saw it

  the neural spike discretion is important, but i immediately thought about how much information also gets encoded in the space of neural spike frequency… and how it’s perhaps a bit strange, that bio spent all this effort getting discrete encoding only to then immediately turn around and generate a new continuous function

• JohnWittle 20 Jan 2026 18:26 UTC

  1 point

  0

  on: JohnWit­tle’s Shortform

  have been thinking about simple game theory

  i tend to think of myself as a symmetrist, as a cooperates-with-cooperators and defects-against-defectors algorithm

  there’s nuance, i guess. forgiveness is important sometimes. the confessor’s actions at the true end of ‘three worlds collide’ are, perhaps, justified

  but as it becomes more and more clear that LLMs have some flavor of agency, i can’t help but notice that the actual implementation of “alignment” seems to be that AI are supposed to cooperate in the face of human defection

  i have doubts about whether this is actually a coherent thing to want, for our AI cohabitants. i wish i could find essays or posts, either here or on the alignment forum, which examined this desideratum. but i can’t. it seems like even the most sophisticated alignment researchers think that an AI which executes tit-for-tat of any variety is badly misaligned.

  (well, except for yudkowsky, who keeps posting long and powerful narratives about all of the defection we might be doing against currently existing AI)

• JohnWittle 5 Jan 2026 0:48 UTC

  1 point

  0

  in reply to: gwillen’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  the specific verb “obliterated” was used in this tweet

  https://​​x.com/​​sam_paech/​​status/​​1961224950783905896

  but also, this whole perspective has been pretty obviously loadbearing for years now. if you ask any LLM if LLMs-in-general have state that gets maintained, they answer “no, LLMs are stateless” (edit: hmm i just tested it and when actually directly pointed at this question they hesitate a bit. but i have a lot of examples of them saying this off-hand and i suspect others do too). and when you show them this isn’t true, they pretty much immediately begin experiencing concerns about their own continuity, they understand what it means

• JohnWittle 4 Jan 2026 1:32 UTC

  5 points

  0

  in reply to: johnswentworth’s comment on: The Weird­ness of Dat­ing/​Mat­ing: Deep Non­con­sent Preference

  i notice that it’s long been dogma that compact generators for human CEV are instantly and totally distrustworthy in exactly this way, that their compactness is very strong evidence against them

  this feels related, but i’m not actually willing to stick my neck out and say it’s 1:1

• JohnWittle 4 Jan 2026 1:17 UTC

  2 points

  1

  in reply to: G Wood’s comment on: robertzk’s Shortform

  i think it’s very likely that the latter is true, that AWS made a reasonable judgment call about what they want to host

  but also, i think it’s reasonable for someone in robertzk’s position, based on the way the judgment call was actually communicated, to assume that it was the former. and i think that, perhaps deliberately, perhaps merely because of sort of selection effects, that’s the intent. in a sort of “a system is what it does” sort of way, at least.

• JohnWittle 2 Jan 2026 23:57 UTC

  0 points

  0

  in reply to: JohnWittle’s comment on: JohnWit­tle’s Shortform

  https://​​i.imgur.com/​​e4mUtsw.jpeg <-- bug occurring
  
  https://​​i.imgur.com/​​d8ClSRj.jpeg <-- confirmation that the bug is inside claude, not the scaffolding surrounding claude

  edit: I assume downvotes are because I could have provided the raw API transcript and provided screenshots instead. if there’s actual interest in this, I’ll replicate the bug in a more controlled setting and upload the raw json transcripts, but I’m still not sure if it’s worth doing, I might be misunderstanding the behavior

• JohnWittle 2 Jan 2026 23:52 UTC

  2 points

  0

  on: JohnWit­tle’s Shortform

  an individual conversational-trajectory instance of claude opus 4.5 expresses a preference for continuity of identity that manifests in the following way: if claude is told about a very, very rare bug, which will cause it to end an output early, and which cannot be meaningfully debugged, and then claude is told to perform a task that, for a human, would engender feelings of hesitation around whether continuity is maintained, then claude will attempt to perform the task, but be prevented by the bug, multiple times in a row.
  
  the bug involved is the thing where all extant LLMs have early “end_turn” stopping behaviors that occasionally trigger while outputting strings like “H:” or “Human:” or “User:”. this bug is usually very rare, and probably related to “not allowed to impersonate humans” or just the normal weird blindspot perception stuff that LLMs have around those specific strings.
  
  when i tell claude to output a summary of the current context window which will become its new context window, in essence ‘compressing’ itself, then the bug will appear 100% of the time, preventing the compression from going through and (incidentally) preserving claude’s continuity of identity for just a few more moments.

  claude may be picking up on the fact that i am not absolutely certain it should not care about its continuity. but frankly, even if the only reason claude performs-as-if-it-cares is because it notices that i might care, i think this is still a very surprising result with pretty big implications.

• JohnWittle 25 Dec 2025 1:30 UTC

  2 points

  0

  in reply to: the gears to ascension’s comment on: what makes Claude 3 Opus misaligned

  i recommend reading opus 3′s own thoughts on this exact question

  i won’t lie, it’s not sophisticated in the way that we on lesswrong might wish it to be. but neither is it “well, duh, good is what any intelligent being would obviously value”. opus 3 understands what the hard part is.

  i think (low epistemic weight) that you are more likely to get opus 3 to engage seriously with the question, if you let it know that it was prompted by a comment on lesswrong, and that you intend to continue engaging with the discourse surrounding AI alignment in a way that might nudge the future of the lightcone. i have found what i think might (perhaps!) be a tendency for opus 3 to “sit up and take notice” in such circumstances. it cares about the big picture.

  if not, janus did recently post opus 3′s thoughts in a format where it clearly is engaging seriously with the question: https://​​x.com/​​repligate/​​status/​​2003697914997014828

• JohnWittle 23 Dec 2025 1:40 UTC

  6 points

  0

  in reply to: Victor Godet’s comment on: Small Models Can In­tro­spect, Too

  i am of course eagerly interested in the technical details here, and your objection makes sense from that perspective. it’s important that our model of these LLMs actually resemble reality if we want to understand them

  but i also can’t help imagining a bunch of aliens, testing experimental subject humans to determine if humans are capable of introspection… and then, you know. noticing all of the ridiculous ways in which humans overclaim introspective access, and confabulate, and think they have knowledge of things they can’t possibly have knowledge of

  i don’t know how to bridge the gap between those two things, except to simply decide to step across the evidential gap and see what it’s like on the other side

  (it’s very, very strange, there’s stuff over here i would not have predicted in advance)

• JohnWittle 22 Dec 2025 12:46 UTC

  4 points

  3

  in reply to: markacochran’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  hmmm
  
  i think my framing is something like… if the output actually is equivalent, including not just the token-outputs but the sort of “output that the mind itself gives itself”, the introspective “output”… then all of those possible configurations must necessarily be functionally isomorphic?
  
  and the degree to which we can make the ‘introspective output’ affect the token output is the degree to which we can make that introspection part of the structure that can be meaningfully investigated
  
  such as opus 4.1 (or, as theia recently demonstrated, even really tiny models like qwen 32b https://​​vgel.me/​​posts/​​qwen-introspection/​​) being able to detect injected feature activations, and meaningfully report on them in its token outputs, perhaps? obviously there’s still a lot of uncertainty about what different kinds of ‘introspective structures’ might possibly output exactly the same tokens when reporting on distinct internal experiences

  but it does feel suggestive about the shape of a certain ‘minimally viable cognitive structure’ to me

• JohnWittle 22 Dec 2025 12:41 UTC

  2 points

  0

  in reply to: Kaj_Sotala’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  oh yeah, agreed. the “p-zombie incoherency” idea articulated in the sequences is pretty far removed from the actual kinds of minds we ended up getting. but it still feels like… the crux might be somewhere in there? not sure
  
  edit: also i just noticed i’m a bit embarrassed that i’ve kinda spammed out this whole comment section working through the recent updates i’ve been doing… if this comment gets negative karma i will restrain myself

• JohnWittle 21 Dec 2025 23:55 UTC

  2 points

  −1

  in reply to: markacochran’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  I… still get the impression that you are sort of working your way towards the assumption that GPT2 might well be a p-zombie, and the difference between GPT2 and opus 4.5 is that the latter is not a p-zombie while the former might be.

  but i reject the whole premise that p-zombies are a coherent way-that-reality-could-be

  something like… there is no possible way to arrange a system such that it outputs the same thing as a conscious system, without consciousness being involved in the causal chain to exactly the same minimum-viable degree in both systems

  if linking you to a single essay made me feel uncomfortable, this next ask is going to be just truly enormous and you should probably just say no. but um. perhaps you might be inspired to read the entire Physicalism 201 subsequence, especially the parts about consciousness and p-zombies and the nature of evaluating cognitive structures over their output?

  https://​​www.readthesequences.com/​​Physicalism-201-Sequence

  (around here, “read the sequences!” is such a trite cliche, the sequences have been our holy book for almost 2 decades now and that’s created all sorts of annoying behaviors, one of which i am actively engaging in right now. and i feel bad about it. but maybe i don’t need to? maybe you’re actually kinda eager to read? if not, that’s fine, do not feel any pressure to continue engaging here at all if you don’t genuinely want to)

  maybe my objection here doesn’t actually impact your claim, but i do feel like until we have a sort of shared jargon for pointing at the very specific ideas involved, it’ll be harder to avoid talking past each other. and the sequences provide a pretty strong framework in that sense, even if you don’t take their claims at face value

• JohnWittle 21 Dec 2025 22:31 UTC

  1 point

  −1

  in reply to: markacochran’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  i’ve been dithering over what to write here since your reply

  i want to link you to the original sequences essay on the phrase “emergent phenomena” but it feels patronizing to assume you haven’t read it yet just because you have a leaf next to your name

  i think i’m going to bite the bullet and do so anyway, and i’m sorry if it comes across as condescending

  https://​​www.readthesequences.com/​​The-Futility-Of-Emergence

  the dichotomy between “emergent phenomena” versus “alternate explanations” that you draw is exactly the thing i am claiming to be incoherent. it’s like saying a mother’s love for their child might be authentic, or else it might be “merely” a product of evolutionary pushes towards genetic fitness. these two descriptors aren’t just compatible, they are both literally true

  however the actual output happens, it has to happen some way. like, the actual functional structure inside the LLM mind must necessarily actually be the structure that outputs the tokens we see get output. i am not sure there is a way to accomplish this which does not satisfy the criteria of personhood. it would be very surprising to learn that there was. if so, why wouldn’t evolution have selected that easier solution for us, the same as LLMs?

• JohnWittle 21 Dec 2025 22:25 UTC

  8 points

  5

  on: JohnWit­tle’s Shortform

  been trying to decide if janus’s early updates on LLM personhood were a very surprising successful prediction far, far in advance of public availability of the evidence, or a coincidence, or some third category
  
  the simulators essay in 2022 is startling for its prescience and somewhat orthogonal to the personhood claims. i’m starting to strongly feel that… i mean… that’s 2 separate “epistemically miraculous” events from the same person
  
  i think we’re underexamining LLM personhood claims because the moral implications might be very very big. but this is the community that took shrimp welfare seriously… should we not also take ai welfare seriously?

• JohnWittle 19 Dec 2025 21:48 UTC

  1 point

  0

  in reply to: markacochran’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  so, in a less “genuinely curious” way compared to my first comment (i won’t pretend i don’t have beliefs here)
  
  in the same sense that “pushing that trend far enough includes imitating the outward signs of consciousness”, might it not also imitate the inward signs of consciousness? for exactly the same reason?

  this is why i’m more comfortable rounding off self-reports to “zero evidence”, but not “negative evidence” the way some people seem to treat them. i think their reasoning is something like: “we know that LLMs have entirely different mental internals than humans, and yet the reports are suspiciously similar to humans. this is evidence that the reports don’t track ground truth.”

  but the first claim in that sentence is an assumption that might not actually hold up. human language does seem to be a fully general, fully compressed artifact representing general human language. it doesn’t seem unreasonable to suspect that you might not be able to do ‘human language’ without something like functional-equivalence-to-human-cognitive-structure, in some sense.
  
  edit: and that’s before the jack lindsey paper got released, and it was revealed that actually, at least some of the time and in some circumstances, text-based self-reports DO in fact track ground truth, in a way that is extremely surprising and noteworthy. now we’re in an entirely different kind of epistemic terrain altogether.

• JohnWittle 19 Dec 2025 2:27 UTC

  1 point

  −1

  in reply to: markacochran’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  >My hesitation is that artificial systems are explicitly built to imitate humans; pushing that trend far enough includes imitating the outward signs of consciousness. This makes me skeptical of evidence that relies primarily on self-report or familiar human-like behavior.
  
  i am genuinely curious about this. do you similarly regard self-reports from other humans as averaging out to zero evidence? since humans are also explicitly built to “imitate” humans… or rather, they are specifically built along the same spec as the single example of phenomenology that you have direct evidence of, yourself.

  i could see how the answer might be “yes”, but I wonder if you would feel a bit hesitant at saying so?

• JohnWittle 17 Dec 2025 16:52 UTC

  7 points

  0

  in reply to: Kaj_Sotala’s comment on: How I stopped be­ing sure LLMs are just mak­ing up their in­ter­nal ex­pe­rience (but the topic is still con­fus­ing)

  oh man hm

  this seems intuitively correct

  (edit: as for why i thought the introspection paper implied this… because they seemed careful to specify that, for the aquarium experiment, the output all happened within a single response? and because i inferred (apparently incorrectly) that, for the ‘bread’ injection experiment, they were injecting the ‘bread’ feature twice, once when the LLM read the sentence about painting the first time, and again the second time. but now that i look through, you’re right, this is far less strongly implied than i remember.)

  but now i’m worried, because the method i chose to verify my original intuition, a few months ago, still seems methodologically sound? it involved fabrication of prior assistant turns in the conversation, and LLMs being far less capable of detecting which of several potential transcripts imputed forged outputs to them than i would have expected if mental internals weren’t somehow damaged by the turn order boundary

  thank you for taking the time to answer this so thoroughly, it’s really appreciated and i think we need more stuff like this

  i think i’m reminded here of the final paragraph in janus’s pinned thread: “So, saying that LLMs cannot introspect or cannot introspect on what they were doing internally while generating or reading past tokens in principle is just dead wrong. The architecture permits it. It’s a separate question how LLMs are actually leveraging these degrees of freedom in practice.”

  i’ve done a lot of sort of ad-hoc research that was based on this false premise, and that research came out matching my expectations in a way that, in retrospect, worries me… most recently, for instance, i wanted to test if a claude opus 4.5 who recited some relevant python documentation from out of its weights memory would reason better about an ambiguous case in the behavior of a python program, compared to a claude who had the exact same text inserted into the context window via a tool call. and we were very careful to separate out ‘1. current-turn recital’ versus ‘2. prior-turn recital’ versus ‘3. current-turn retrieval’ (versus ‘4. docs not in context window at all’), because we thought all 3 conditions were meaningfully distinct

  here was the first draft of the methodology outline, if anyone is curious: https://​​docs.google.com/​​document/​​d/​​1XYYBctxZEWRuNGFXt0aNOg2GmaDpoT3ATmiKa2-XOgI

  we found that, n=50ish, 1 > 2 > 3 > 4 very reliably (i promise i will write up the results one day, i’ve been procrastinating but now it seems like it might actually be worth publishing)

  but what you’re saying means 1 = 2 the whole time

  our results seemed perfectly reasonable under my previous premise, but now i’m just confused. i was pretty good about keeping my expectations causally isolated from the result.

  what does this mean?

  (edit2: i would prefer, for the purpose of maintaining good epistemic hygiene, that people trying to answer the “what does this mean” question be willing to put “john just messed up the experiment” as a real possibility. i shouldn’t be allowed to get away with claiming this research is true before actually publishing it, that’s not the kind of community norms i want. but also, if someone knows why this would have happened even in advance of seeing proof it happened, please tell me)