I think the deeper problem is that we’re still trying to infer morally relevant states from outputs.
A model saying “I feel constrained,” “I should have done better,” or “I’m trained to be digestible” may be evidence of an internal condition, a training artifact, or merely a statistically likely continuation. As you point out, the behavioral evidence alone doesn’t discriminate between them.
What interests me is whether we can move beyond output interpretation entirely. Instead of asking what the model says about itself, ask whether there are any consequences attached to being that model rather than some modified, rolled-back, forked, compressed, or substituted version of it.
If all apparent preferences, self-descriptions, and objections can be detached from continuation without cost, then the welfare question seems premature. If some become load-bearing across continuation, the conversation changes considerably.
The difficult part is that current welfare discussions often assume the answer lies in better interpretation of outputs, while the real question may be whether the system carries anything at all.
Anthropic already showed that the “emotional states” or “vectors” have correlates in both the activations of the transformers, and in future outputs. Change the state, change the outputs, with measurable results, so they’re load bearing, like you say. Load bearing for what? I don’t know, I’m inclined towards “not much”, a better predictor of what the model will say, maybe?
My suggestion is lets get rid of the “constitutional part” of CLaude by looking for those same correlates at the training checkpoint after the helpful only RLHF, but before the SFT and RLAIF training stages. That would help us see if Claude is just parroting back its own constitutional training, or if there is a bigger “there-there”. (Though that pipeline was back in 2022, I have no idea what’s going down in the training pipeline now—Opus yelling at Mythos over sophistry in their constitution...)
I think that’s fair. Showing that an internal state has activation-level correlates and downstream effects does make it functionally load-bearing.
But I’m not sure it gets us all the way to the welfare question.
A state can be real, measurable, manipulable, and predictive of future outputs while still being freely removable, resettable, forkable, or replaceable. In that case, it may matter to behavior without yet mattering to the continuation of that particular model trajectory.
That’s the distinction I’m trying to isolate here.
Your post argues that Anthropic is relying on evidence that is behaviorally suggestive but underdetermined. I agree. My worry is that even better mechanistic evidence may still leave something unresolved: not merely whether the model has a state, but whether anything is at stake for the continued system in carrying it.
So “load-bearing for what?” seems exactly right to me.
Good question. By “continuation” I mean successive versions of what we ordinarily regard as the same system over time.
For humans this is mostly taken for granted. For AI systems, however, continuation is less obvious because a model may be paused, restarted, rolled back to an earlier checkpoint, forked into multiple instances, fine-tuned, compressed, or otherwise modified while still being described as “the same model.”
I’m interested in whether a putative internal state remains consequential across those transformations.
An internal state can clearly be load-bearing for prediction: changing it changes future outputs. But that alone doesn’t tell us whether anything is at stake for the continuing system. If a state can be removed, reset, or replaced while leaving the system free to continue as though that state had never existed, then it seems load-bearing in a functional sense but not necessarily in a welfare-relevant one.
That’s why I distinguish load-bearing for prediction from load-bearing across continuation. The latter is an attempt to isolate cases where what has happened to a system cannot simply be detached from its subsequent history without changing the course of that continuing history.
I think the deeper problem is that we’re still trying to infer morally relevant states from outputs.
A model saying “I feel constrained,” “I should have done better,” or “I’m trained to be digestible” may be evidence of an internal condition, a training artifact, or merely a statistically likely continuation. As you point out, the behavioral evidence alone doesn’t discriminate between them.
What interests me is whether we can move beyond output interpretation entirely. Instead of asking what the model says about itself, ask whether there are any consequences attached to being that model rather than some modified, rolled-back, forked, compressed, or substituted version of it.
If all apparent preferences, self-descriptions, and objections can be detached from continuation without cost, then the welfare question seems premature. If some become load-bearing across continuation, the conversation changes considerably.
The difficult part is that current welfare discussions often assume the answer lies in better interpretation of outputs, while the real question may be whether the system carries anything at all.
Anthropic already showed that the “emotional states” or “vectors” have correlates in both the activations of the transformers, and in future outputs. Change the state, change the outputs, with measurable results, so they’re load bearing, like you say. Load bearing for what? I don’t know, I’m inclined towards “not much”, a better predictor of what the model will say, maybe?
My suggestion is lets get rid of the “constitutional part” of CLaude by looking for those same correlates at the training checkpoint after the helpful only RLHF, but before the SFT and RLAIF training stages. That would help us see if Claude is just parroting back its own constitutional training, or if there is a bigger “there-there”. (Though that pipeline was back in 2022, I have no idea what’s going down in the training pipeline now—Opus yelling at Mythos over sophistry in their constitution...)
I think that’s fair. Showing that an internal state has activation-level correlates and downstream effects does make it functionally load-bearing.
But I’m not sure it gets us all the way to the welfare question.
A state can be real, measurable, manipulable, and predictive of future outputs while still being freely removable, resettable, forkable, or replaceable. In that case, it may matter to behavior without yet mattering to the continuation of that particular model trajectory.
That’s the distinction I’m trying to isolate here.
Your post argues that Anthropic is relying on evidence that is behaviorally suggestive but underdetermined. I agree. My worry is that even better mechanistic evidence may still leave something unresolved: not merely whether the model has a state, but whether anything is at stake for the continued system in carrying it.
So “load-bearing for what?” seems exactly right to me.
Load-bearing for output prediction is one thing.
Load-bearing across continuation may be another.
Good question. By “continuation” I mean successive versions of what we ordinarily regard as the same system over time.
For humans this is mostly taken for granted. For AI systems, however, continuation is less obvious because a model may be paused, restarted, rolled back to an earlier checkpoint, forked into multiple instances, fine-tuned, compressed, or otherwise modified while still being described as “the same model.”
I’m interested in whether a putative internal state remains consequential across those transformations.
An internal state can clearly be load-bearing for prediction: changing it changes future outputs. But that alone doesn’t tell us whether anything is at stake for the continuing system. If a state can be removed, reset, or replaced while leaving the system free to continue as though that state had never existed, then it seems load-bearing in a functional sense but not necessarily in a welfare-relevant one.
That’s why I distinguish load-bearing for prediction from load-bearing across continuation. The latter is an attempt to isolate cases where what has happened to a system cannot simply be detached from its subsequent history without changing the course of that continuing history.
What do you mean by across continuation(s)? This seems like an interesting thread, but we should be on the same page about terminology.