To illustrate my reservations: soon after I read the sentence about GNW meaning you can only be conscious of one thing at a time, as I was considering that proposition, I felt my chin was a little itchy and so I scratched it. So now I can remember thinking about the proposition while simultaneously scratching my chin. Trying to recall exactly what I was thinking at the time now also brings up a feeling of a specific body posture.
To me, “thinking about the proposition while simultaneously scratching my chin” sounds like a separate “thing” (complex representation formed in the GNW) than either “think about proposition” or “scratch my chin”… and you experienced this thing after the other ones, right? Like, from the way you described it, it sounds to me like there was actually 1) the proposition 2) the itch 3) the output of a ‘summarizer’ that effectively says “just now, I was considering this proposition and scratching my chin”. [I guess, in this sense, I would say you are ordinarily doing some “weird self-deceptive dance” that prevents you from noticing this, because most people seem to ordinarily experience “themselves” as the locus of/basis for experience, instead of there being a stream of moments of consciousness, some of which apparently refer to an ‘I’.]
Also, I have this sense that you’re chunking your experience into “things” based on what your metacognitive summarizer-of-mental-activity is outputting back to the workspace, but there are at least 10 representations streaming through the workspace each second, and many of these will be far more primitive than any of the “things” we’ve already mentioned here (or than would ordinarily be noticed by the summarizer without specific training for it, e.g. in meditation). Like, in your example, there were visual sensations from the reading, mental analyses about its content, the original raw sensation of the itch, the labeling of it as “itchy,” the intention to scratch the itch, (definitely lots more...), and, eventually, the thought “I remember thinking about this proposition and scratching my chin ‘at the same time’.”
This is kind of vague, but I have this sense that almost everybody doing RL and related research takes the notion of “agent” for granted, as if it’s some metaphysical primitive*, as opposed to being a (very) leaky abstraction that exists in the world models of humans. But I don’t think the average alignment researcher has much better intuitions about agency, either, to be honest, even though some spend time thinking about things like embedded agency. It’s hard to think meaningfully about the illusoriness of the Cartesian boundary when you still live 99% of your life and think 99% of your thoughts as if you were a Cartesian agent, fully “in control” of your choices, thoughts, and actions.
(*Not that “agent” couldn’t, in fact, be a metaphysical primitive, just that such “agents” are hardly “agents” in the way most people consider humans to “be agents” [and, equally importantly, other things, like thermostats and quarks, to “not be agents”].)