I… still get the impression that you are sort of working your way towards the assumption that GPT2 might well be a p-zombie, and the difference between GPT2 and opus 4.5 is that the latter is not a p-zombie while the former might be.
but i reject the whole premise that p-zombies are a coherent way-that-reality-could-be
something like… there is no possible way to arrange a system such that it outputs the same thing as a conscious system, without consciousness being involved in the causal chain to exactly the same minimum-viable degree in both systems
if linking you to a single essay made me feel uncomfortable, this next ask is going to be just truly enormous and you should probably just say no. but um. perhaps you might be inspired to read the entire Physicalism 201 subsequence, especially the parts about consciousness and p-zombies and the nature of evaluating cognitive structures over their output?
(around here, “read the sequences!” is such a trite cliche, the sequences have been our holy book for almost 2 decades now and that’s created all sorts of annoying behaviors, one of which i am actively engaging in right now. and i feel bad about it. but maybe i don’t need to? maybe you’re actually kinda eager to read? if not, that’s fine, do not feel any pressure to continue engaging here at all if you don’t genuinely want to)
maybe my objection here doesn’t actually impact your claim, but i do feel like until we have a sort of shared jargon for pointing at the very specific ideas involved, it’ll be harder to avoid talking past each other. and the sequences provide a pretty strong framework in that sense, even if you don’t take their claims at face value
No, no. I appreciate it. So, it seems like even if consciousness is physical and non-mysterious, evidence thresholds could differ radically between evolved biological systems and engineered imitators.
I think we may be talking past each other a bit. I’m not committed to p-zombies as a live metaphysical possibility, and I’m not claiming that “emergent” is an explanation.
My uncertainty is narrower: even if I grant physicalism and reject philosophical zombies, it still seems possible for multiple internal causal organizations to generate highly similar linguistic behavior. If so, behavior alone may underdetermine phenomenology for artificial systems in a way it doesn’t for humans.
That’s why I keep circling back to discriminants that are hard to get “for free” from imitation: intervention sensitivity, non-linguistic control loops, or internal-variable dependence that can’t be cheaply faked by next-token prediction.
i think my framing is something like… if the output actually is equivalent, including not just the token-outputs but the sort of “output that the mind itself gives itself”, the introspective “output”… then all of those possible configurations must necessarily be functionally isomorphic?
and the degree to which we can make the ‘introspective output’ affect the token output is the degree to which we can make that introspection part of the structure that can be meaningfully investigated
such as opus 4.1 (or, as theia recently demonstrated, even really tiny models like qwen 32b https://vgel.me/posts/qwen-introspection/) being able to detect injected feature activations, and meaningfully report on them in its token outputs, perhaps? obviously there’s still a lot of uncertainty about what different kinds of ‘introspective structures’ might possibly output exactly the same tokens when reporting on distinct internal experiences
but it does feel suggestive about the shape of a certain ‘minimally viable cognitive structure’ to me
there is no possible way to arrange a system such that it outputs the same thing as a conscious system, without consciousness being involved in the causal chain to exactly the same minimum-viable degree in both systems
GPT-2 doesn’t have the same outputs as the kinds of systems we know to be conscious, though! The concept of a p-zombie is about someone who behaves like a conscious human in every way that we can test, but still isn’t conscious. I don’t think the concept is applicable to a system that has drastically different outputs and vastly less coherence than any of the systems that we know to be conscious.
oh yeah, agreed. the “p-zombie incoherency” idea articulated in the sequences is pretty far removed from the actual kinds of minds we ended up getting. but it still feels like… the crux might be somewhere in there? not sure
edit: also i just noticed i’m a bit embarrassed that i’ve kinda spammed out this whole comment section working through the recent updates i’ve been doing… if this comment gets negative karma i will restrain myself
I agree with you on a lot of points, I’m just saying that text-based responses to prompts are an imperfect test for phenomenology in the case of large language models.
I think the key step still needs an extra premise. “Same external behavior (even including self-reports) ⇒ same internal causal organization” doesn’t follow in general; many different internal mechanisms can be behaviorally indistinguishable at the interface, especially at finite resolution. You, me, and every other human mind only ever observe systems at a limited “resolution” or “frame rate.” If, as observers, we had a much lower resolution or frame rate we might very well think that GPT2 is indistinguishable from human output.
To make the inference go through, you’d need something like: (a) consciousness just is the minimal functional structure required for those outputs, or (b) the internal-to-output mapping is constrained enough to be effectively one-to-one. Otherwise, we’re back in an underdetermination problem, which is why I find the intervention-based discriminants so interesting.
I… still get the impression that you are sort of working your way towards the assumption that GPT2 might well be a p-zombie, and the difference between GPT2 and opus 4.5 is that the latter is not a p-zombie while the former might be.
but i reject the whole premise that p-zombies are a coherent way-that-reality-could-be
something like… there is no possible way to arrange a system such that it outputs the same thing as a conscious system, without consciousness being involved in the causal chain to exactly the same minimum-viable degree in both systems
if linking you to a single essay made me feel uncomfortable, this next ask is going to be just truly enormous and you should probably just say no. but um. perhaps you might be inspired to read the entire Physicalism 201 subsequence, especially the parts about consciousness and p-zombies and the nature of evaluating cognitive structures over their output?
https://www.readthesequences.com/Physicalism-201-Sequence
(around here, “read the sequences!” is such a trite cliche, the sequences have been our holy book for almost 2 decades now and that’s created all sorts of annoying behaviors, one of which i am actively engaging in right now. and i feel bad about it. but maybe i don’t need to? maybe you’re actually kinda eager to read? if not, that’s fine, do not feel any pressure to continue engaging here at all if you don’t genuinely want to)
maybe my objection here doesn’t actually impact your claim, but i do feel like until we have a sort of shared jargon for pointing at the very specific ideas involved, it’ll be harder to avoid talking past each other. and the sequences provide a pretty strong framework in that sense, even if you don’t take their claims at face value
No, no. I appreciate it. So, it seems like even if consciousness is physical and non-mysterious, evidence thresholds could differ radically between evolved biological systems and engineered imitators.
I think we may be talking past each other a bit. I’m not committed to p-zombies as a live metaphysical possibility, and I’m not claiming that “emergent” is an explanation.
My uncertainty is narrower: even if I grant physicalism and reject philosophical zombies, it still seems possible for multiple internal causal organizations to generate highly similar linguistic behavior. If so, behavior alone may underdetermine phenomenology for artificial systems in a way it doesn’t for humans.
That’s why I keep circling back to discriminants that are hard to get “for free” from imitation: intervention sensitivity, non-linguistic control loops, or internal-variable dependence that can’t be cheaply faked by next-token prediction.
hmmm
i think my framing is something like… if the output actually is equivalent, including not just the token-outputs but the sort of “output that the mind itself gives itself”, the introspective “output”… then all of those possible configurations must necessarily be functionally isomorphic?
and the degree to which we can make the ‘introspective output’ affect the token output is the degree to which we can make that introspection part of the structure that can be meaningfully investigated
such as opus 4.1 (or, as theia recently demonstrated, even really tiny models like qwen 32b https://vgel.me/posts/qwen-introspection/) being able to detect injected feature activations, and meaningfully report on them in its token outputs, perhaps? obviously there’s still a lot of uncertainty about what different kinds of ‘introspective structures’ might possibly output exactly the same tokens when reporting on distinct internal experiences
but it does feel suggestive about the shape of a certain ‘minimally viable cognitive structure’ to me
GPT-2 doesn’t have the same outputs as the kinds of systems we know to be conscious, though! The concept of a p-zombie is about someone who behaves like a conscious human in every way that we can test, but still isn’t conscious. I don’t think the concept is applicable to a system that has drastically different outputs and vastly less coherence than any of the systems that we know to be conscious.
oh yeah, agreed. the “p-zombie incoherency” idea articulated in the sequences is pretty far removed from the actual kinds of minds we ended up getting. but it still feels like… the crux might be somewhere in there? not sure
edit: also i just noticed i’m a bit embarrassed that i’ve kinda spammed out this whole comment section working through the recent updates i’ve been doing… if this comment gets negative karma i will restrain myself
I agree with you on a lot of points, I’m just saying that text-based responses to prompts are an imperfect test for phenomenology in the case of large language models.
I think the key step still needs an extra premise. “Same external behavior (even including self-reports) ⇒ same internal causal organization” doesn’t follow in general; many different internal mechanisms can be behaviorally indistinguishable at the interface, especially at finite resolution. You, me, and every other human mind only ever observe systems at a limited “resolution” or “frame rate.” If, as observers, we had a much lower resolution or frame rate we might very well think that GPT2 is indistinguishable from human output.
To make the inference go through, you’d need something like: (a) consciousness just is the minimal functional structure required for those outputs, or (b) the internal-to-output mapping is constrained enough to be effectively one-to-one. Otherwise, we’re back in an underdetermination problem, which is why I find the intervention-based discriminants so interesting.