I would honestly be very surprised if you actually kept track of every visual detail verbally.
Sorry, I didn’t really explain this very well—I’m not tracking them verbally. In fact, most of the time I don’t tend to represent my thoughts verbally either (something else that seems to surprise people when we discuss how we think), they exist in the same state that my spacial representations of things do—at least until I need them in a verbal form (such as when I’m trying to decide on the wording of something I’m going to say or write). It’s hard for me to describe the state my thoughts take to someone else, as there’s no analogue outside of my own head. When I said:
I can remember details of appearance, but they take the form of ‘has ~5cm, dark straight hair’, rather than a picture of how said hair might look.
What I meant is that there are some sort of symbols in my mind that represent these concepts being activated, but in a more abstract way than by using the words that describe them, or by picturing the images that make me think of them. It feels a little like there’s a more abstract layer that sits on top of my visual and verbal systems, and this is where I do my thinking and imagining. If I need to, I can ‘bring my thought down’ to these parts of my brain (like when I’m deciding what to write, or how to draw something), but it’s not the default case.
It seems to me like your visual cortex is processing the “picture” but for some reason you aren’t experiencing it directly...
I guess this could be a possibility, but I do experience the “picture” to a degree if I actually make an effort to visualise it, like when I’m trying to draw a scene from my imagination. It’s not really anything like what I see when I’m actually looking at something, but I assume that’s the case for everyone. For example, if I try to do a puzzle that involves picturing something from a different angle, I’m able to, but it takes a conscious effort. The process I use for this doesn’t seem to get involved when I’m imagining a scene from a book (unless I’m doing something like trying to picture a scene from someone else’s point of view, at which point I have to stop reading briefly while I build up a picture).
I’m not an expert either, but from what I’ve read on the subject, most of the neocortex does work like this. The architecture used in the visual cortex is largely the same as that used in the rest of the cortex, with some minor variations. This is suggested by the fact that people who lose an area of their neocortex are often able to recover, with another area filling in for it. I’m on a phone, so I can’t go into as much detail as I’d like, but I recommend investigating the work of Mountcastle, and more recently Markram.
Edit: On Intelligence by Jeff Hawkins explains this principle in more depth, it’s an interesting read.