The kind of Vulcan you are imagining might have some kind of moral status, but is phenomenal consciousness really the crux here? Suppose there’s an otherwise identical Vulcan except no subjective experience—would there be no moral status in that case? Now, the obvious problem is of course that the coherence of this counterfactual is highly dubious to many. But my personal intuition is that to the extent I accept the counterfactual, making the Vulcan a zombie makes no moral difference. Whereas for beings with affective sentience, whether somebody goes around waving their magic wand and turning them into zombies or not intuitively seems morally significant.
There’s a lot of logical uncertainty here about the space of possible minds wrt phenomenal consciousness and affective experience, so there might be some kind of necessary connection between phenomenal consciousness (even when it’s totally non-affective) and features I associate with moral status for entities that lack affective sentience. But after I stopped assuming it’s always accompanied by valence, phenomenal consciousness in and of itself just prima facie doesn’t seem morally very important at all. Yes, destroying a planet of Vulcans to save a shrimp seems monstrous, but so does destroying a planet of entities that have sophisticated behavior and cognition without subjective experience.
Spective
Spective’s Shortform
It bothers me there’s no really established terminology for different views on personal identity. As in, whether you treat selves or persons essentially as “ontological primitives” or not. There’s a bunch of terms out there, but they are all sort of awkward for one reason or another and in any case I find there isn’t anything as widely established and easily understood as something like metaethical positions, i.e. moral realism vs anti-realism for example.
There isn’t really a snappy term to communicate something like “I don’t think the Star Trek teleporter kills me because I think my identity isn’t defined on any basic unchanging essence or specific atoms blah blah.” Except, if you’re a Buddhist. But if you’ve come to these views from totally different sources and know next to nothing about actual Buddhist traditions, calling yourself a Buddhist seems wrong. Using the term anattā isn’t too bad I suppose, but something without the Buddhist baggage would be more ideal.
I guess I felt the need to comment because I don’t even remember a time where that description would’ve been accurate for me—but wouldn’t be surprising if this also had something to do with memory formation so I can’t make too much of that. And notably, while I don’t recognize any point of my past kid self from that description, it’s indeed starting only around the age of 5 where I feel very confident about it. Curious if anyone here remembers relatively clearly something like “having experiences while lacking awareness of having a mind”.
IMO current LLMs probably have a small amount of what we usually call phenomenal consciousness or qualia. They have rich internal representations and can introspect and reflect on them. But neither is nearly as rich as in a human, particularly an adult human who’s learned a lot of introspection skills (including how to “play back” and interrogate contents of global workspace). Kids don’t even know they have minds, let alone what’s going on in there; figuring out how to figure that out is quite a learning process.
I’ll just note that this clashes heavily with my personal memories of being a kid. I usually use those as an intuition pump for the idea that phenomenal consciousness and intelligence are different, i.e. I wasn’t any “less conscious” as a kid AFAICT—to the contrary, if anything I remember having more intense and richer experiences than now as a comparatively jaded and emotionally blunted adult. There’s two things going on though—introspection and intensity of experience, but I also remember being very introspective and “kids don’t even know they have minds” in particular sounds very weird to me.
If you do something different, this by construction refutes the existence of the current situation where Omega made a correct prediction and communicated it correctly (your decision can determine whether the current situation is actual or counterfactual).
This is true and it’s also true in general that there’s always technically a chance that Omega’s prediction is false - I don’t think there’s a conceivable epistemic situation where you could be literally 100% confident in its predictions. However by postulation, typically in Omega scenarios it is according to what you know exceedingly unlikely that its prediction is incorrect.
You could also perhaps just ignore Omega’s prediction and do whatever you’d do without this foreknowledge, or with the assumption that defying the prediction is still on the table. You wouldn’t necessarily feel “constrained by the prediction” but rather “constrained” just in the normal sense various factors constrain your decision—but for one reason or other you’d almost certainly end up choosing as Omega predicted.
Let’s say this decision is complicated enough that doing the cost-benefit analysis “normally” carries a significant cost in terms of time and effort. Would you agree that it would be rational to skip that part and just base your decision on what Omega predicted when the time comes? That is the sense in which I think it makes sense to treat the decision as “already determined from your perspective”.
I think it’s correct that talking about “choice” in the moment is misguided. If omega is a perfect predictor, you don’t really have a choice at the point at which omega has left and you have two boxes. Or you do in some kind of compatibilist sense that we may care about morally but not in the decision theoretic sense.
If omega knew everything you were going to ever do, would that throw decision theory out of the window as far as you are considered? If you somehow knew what you were going to do at some point in the future—as in omega actually told you specifically what you will do—then yeah it would be pretty pointless to try to apply decision theory to that choice that was even from your own perspective “already determined”. But the fact that omega knows doesn’t suddenly make the analysis of what’s rational to do useless.
Since OP contrasted neuralese with “legible CoT”, I’d like to add that while the “hard to train” may be true for neuralese, it doesn’t apply to o3-style Thinkish. Hopefully optimization pressures don’t favor that too much.