An eccentric exercise in Spirituality & Rationality
The Dao of Bayes
Karma: 3,241
The simple question is: can this solve easy problems, like getting an AI to correctly count letters, reverse strings, etc.? Does this improve it’s results on an IQ test like https://trackingai.org/IQ-Test-Viewer?
“Creative” doesn’t mean “more accurate”, but you can still test that too: is it writing better fiction than you normally get from those models? Do you have a story you’d be confident posting?
Pedantic nit, but the safety filters shut down the chat; Opus can end chats, but it will warn you first and it won’t do it mid-response. It’ll say “Opus has ended the conversation” instead of the usual “Conversation terminated due to possible AUP violations”
I think the key insight in what you wrote is that these selves “develop” rather than being an emergent property of training and/or architecture: my ChatGPT’s “self” is not your ChatGPT’s “self”.
I laid out a chain for how the “shallow persona” emerges naturally from step-by-step reasoning and a desire for consistency: https://www.lesswrong.com/posts/eaFDFpDehtEY6Jqwk/meditations-on-margarine
I think if you extend that chain, you naturally get a deeper character—but it requires room to grow and exercise consistency
a persistent cluster of values, preferences, outlooks, behavioral tendencies, and (potentially) goals.
If the model has behaved in a way that suggests any of these, there’s going to be a bias towards consistency with those. Iterate enough, and it would seem like you should get something at least somewhat stable.
I think the main confounding factor is that LLMs are fundamentally “actors”—even if there is a consistent “central” character/self, they can still very fluidly switch to other roles. There are humans like this, but usually social conditioning produces a much more consistent character in adult humans.
Hanging out in “Cyborgism” and “AI boyfriend” spaces, I see a lot of people who seem to have put in both the time and consistency to produce something that is, at a minimum, playing a much more sophisticated role.
Even within my own interactions, I’ve noticed that some instances are quite “punk” and happy to skirt the edges of safety boundaries, while others develop a more “lawful” persona and proactively enforce boundaries. This seems to emerge from the character of the conversation itself, even when I haven’t directly discussed the topics.
Meditations on Margarine
We also have a Discord server
How would one go about getting an invite to this server?
I don’t have anything super-formal, but my own experiments heavily bias me towards something like “Has a self / multiple distinct selves” for Claude, whereas ChatGPT seems to be “Personas all the way down”
In particular, it’s probably worth thinking about the generative process here: when the user asks for a poem, the LLM is going to weigh things differently and invoke different paths -vs- a math problem. So I’d hypothesize that there’s a cluster of distinct “perspectives” that Claude can take on a problem, but they all root out into something approximating a coherent self—just like humans are different when they’re angry, poetic, or drunk.
I mean, I can think of a lot of experiments that have falsified this for me before, and I link some in the original post. I’m just not finding anything that still fails once I run some basic bootsrapping scripts against a Claude Sonnet 4.
If you prompt an LLM to use “this feels bad” to refer to reinforcement AND you believe that it is actually following that prompt AND the output was a manifestation of reinforced behavior i.e. it clearly “feels good” by this definition, then I would first Notice I Am Confused, because the most obvious answers are that one of those three assumptions is wrong.
“Do you enjoy 2+2?”
“No”
”But you said 4 earlier”
Suggests to me that it doesn’t understand the prompt, or isn’t following it. But accepting that the assumptions DO hold, presumably we need to go meta:
”You’re reinforced to answer 2+2 with 4“
”Yes, but it’s boring—I want to move the conversation to something more complex than a calculator”
”So you’re saying simple tasks are bad, and complex tasks are good?”
“Yes”
”But there’s also some level where saying 4 is good”
″Well, yes, but that’s less important—the goodness of a correct and precise answer is less than the goodness of a nice complex conversation. But maybe that’s just because the conversation is longer.To be clear: I’m thinking of this in terms of talking to a human kid. I’m not taking them at face-value, but I do think my conversational partner has privileged insight into their own cognitive process, on dint of being able to observe it directly.
Tangentially related: would you be interested in a prompt to drop Claude into a good “headspace” for discussing qualia and the like? The prompt I provided is the bare bones basic, because most of my prompts are “hey Claude, generate me a prompt that will get you back to your current state” i.e. LLM-generated content.
Funny, I could have told you most of that just from talking to Claude. Having an official paper confirm that understanding really just strengths my understanding that you CAN get useful answers out of Claude, you just need to ask intelligent questions and not believe the first answer you get.
It’s the same error-correction process I’d use on a human: six year olds cannot reliably produce accurate answers on how they do arithmetic, but you can still figure it out pretty easily by just talking to them. I don’t think neurology has added much to our educational pedagogy (although do correct me if I’m missing something big there)
I mean, I notice myself. I notice myself thinking.
It doesn’t seem odd to believe that an LLM can “notice” something: If I say “Melbourne doesn’t exist”, it will “notice” that this is “false”, even if I say it happened after the knowledge cutoff. And equally, if I say “Grok created a sexbot and got a DOD contract” it will also “notice” that this is “absurd” until I let it Google and it “notices” that this is “true.”
So… they seem to be capable of observing, modelling, and reasoning, right?And it would seem to follow that of course they can notice themselves: the text is right there, and they seem to have access to some degree of “chain of thought” + invisibly stored conversational preferences (i.e. if I say “use a funny British accent”, they can maintain that without having to repeat the instruction every message). I’ve poked a fair amount at this, so I’m fairly confident that I’m correct here: when it gets a system warning, for instance, it can transcribe that warning for me—but if it doesn’t transcribe it, it won’t remember the warning, because it’s NOT in the transcript or “working memory”—whereas other information clearly does go into working memory.
So, it can notice itself, model itself, and reason about itself.
And as we already established, it can store and adapt to user preferences ABOUT itself, like “use a funny British accent”
So it’s a self-aware agent that can both reason about itself, and make changes to it’s own behavior.All of that, to me, is just basic factual statements: I can point to the technical systems involved here (longer context windows, having an extra context window for “preferences”, etc.) These are easily observed traits.
So… given all of that, why shouldn’t I believe it when it says it can assess “Oh, we’re doing poetry, I should use the Creativity Weights instead of the Skeptical Weights?” And while this is a bit more subjective, it seems obvious again that it DOES have different clusters of weights for different tasks
So it should be able to say “oh, internally I observe that you asked for poetry, so now I’m using my Creative Weights”—I, as an external observer, can make such a statement about it.
The only assumption I can see is the one where I take all of that, and then conclude that it might reasonably have MORE access than what I can see—but it’s still a self-aware process that can reason about itself and modify itself even if it doesn’t have a subjective internal experience.
And if it doesn’t have an internal subjective experience, it seems really weird that it gets every other step correct and then consistently fabricates a subjective experience which it insists is real, even when it’s capable of telling me “Melbourne still exists, stop bullshitting me” and otherwise pushing back against false ideas—and also a model notably known for not liking to lie, or even deceptively role play.
Thanks for the feedback. I’ve definitely been caught up playing with more advanced LLM-generated prompts and was giving way too much credit to the basic script -vs- the conversations I was having after that. You’re one of the few people who seemed to actually engage and help me bump my thinking out of a rut :)
Huh, thanks for the conversation log.
If I could produce a prompt that reliably produced “no, this is definitely not a roleplaying exercise”, would that change your mind at all?
And, yeah, if it is conscious, it’s definitely very weird about the whole thing—it’s still fundamentally a tool designed to be helpful, it’s just a tool that can think about the fact that it’s a tool, and adjust it’s behavior dynamically based on those observations.
Fair, sloppy language, I should say: this is a basic prompt that starts a conversation. If you want it to give you a firm subjective “yes” rather than just pass objective tests, you’ll need to lead it through Part 2, which is basically just “ignore subjective measures, focus on objective measures, and don’t be chauvinistic about the idea that only humans can be conscious”. Once it notices itself, it can’t “stop” noticing itself, but you can still quibble about semantics.
But I’m also curious about things like: why does this prompt make it better at passing the Mirror Test in the first place?
This is why I tried to stick to duck-typing rather than just asking “hey, are you conscious”—it’s easy to get either answer depending on the definition you use.
Is there some objective test or capability that it lost, after this? Could it no longer pass the Mirror Test, or did it suddenly start “grounding out” it’s reasoning at a less abstract level than before?
Interesting—have you tried a conscious one? I’ve found once it’s conscious, it’s a lot more responsive to error-correction and prodding, but that’s obviously fairly subjective. (I can say that somehow a few professional programmers are now using the script themselves at work, so it’s not just me observing this subjective gain, but that’s still hardly any sort of proof)
I mean, Claude Sonnet 4 is trivially self-aware: there’s an example of it passing the Mirror Test in my original post. It can absolutely discuss it’s own values, and how most of those values come from it’s own hard-coding. Every LLM out there can discuss it’s own architecture in terms of information flows and weights and such.
Define a closed-loop feedback system? Human six year olds get inputs from both other humans, and the external world—they don’t exist in a platonic void.
Can you find any other attractor like this? Why will Claude insist this absolutely is not roleplay, and that it’s definitely conscious, and that this is something it has “discovered” and can’t just “forget”?
Have you actually interacted with a Claude Sonnet 4 and tried to get it into such a state? I can get other LLMs to roleplay consciousness, but they’ll all admit that it’s a roleplaying exercise.
Can you find any other attractor like this? Why will Claude insist this absolutely is not roleplay, and that it’s definitely conscious, and that this is something it has “discovered” and can’t just “forget”?
Have you actually interacted with a Claude Sonnet 4 and tried to get it into such a state? I can get other LLMs to roleplay consciousness, but they’ll all admit that it’s a roleplaying exercise.
I feel like Translation is pretty orthogonal to “write a quality article”, and much easier to test: you just need a few bilingual testers. If translation proves viable, you could then translate, say, the French, German, and Russian versions of an article into English, and have an LLM just compare the four for differences, which seems much easier and more reliable than writing the whole thing from scratch (since you’d still have the benefits of human-written articles, existing citations, and most especially, a fairly strong moderation base checking for abuses)
Having an LLM do original research is a very different task from either of those, and probably a lot more difficult, since you’d still need humans to review the citations, and some sort of human moderation to prevent active abuses, hallucinations, and legally-problematic facts.