basically: what LLMs are “thinking” when they answer.
I found this interesting:
In humans, most of the brain’s processing is not conscious—we don’t deliberately think about parsing grammar while reading, or balancing our bodies while walking. Similarly, we found that most of Claude’s processing doesn’t involve its J-space. It turns out that the J-space holds only a few dozen concepts at a time, and accounts for less than a tenth of the overall activity in Claude’s internal processing. So what is all the rest of the neural network doing? [...]
It turns out the rest of the network can do quite a lot. Without its J-space, Claude speaks fluently, classifies sentiment, answers multiple-choice questions, and pulls facts out of passages roughly as well as before. What it loses, though, are the tasks that require some higher-order thinking: multi-step reasoning drops to near zero, and summarization and rhyming poetry-writing performance fall below the level of a much smaller, intact model.
Anthropic should do this experiment (“think of a number...”) and see what shows up in the J space. My prediction would be that it wouldn’t show a specific number (e.g. “4″) but the concept “number” or “placeholder for a number”.
Maybe someone can try this experiment with an Open model?
Related: Anthropic’s A global workspace in language models
basically: what LLMs are “thinking” when they answer.
I found this interesting:
Anthropic should do this experiment (“think of a number...”) and see what shows up in the J space. My prediction would be that it wouldn’t show a specific number (e.g. “4″) but the concept “number” or “placeholder for a number”.
Maybe someone can try this experiment with an Open model?