I agree that what LLMs are doing is something like what you described. But if those types of circuits can produce reasoning, which apparently they at least sort of can, the sum total behavior may be very different in kind from the local behavior that produces it.
In fact, I’d say this is also roughly what is going on with the human brain. Some pretty janky mechanisms produce each thought; but the thoughts work together to get some impressive stuff done. Humans would be nothing without system 2, what you call reasoning and don’t address.
I agree that what LLMs are doing is something like what you described. But if those types of circuits can produce reasoning, which apparently they at least sort of can, the sum total behavior may be very different in kind from the local behavior that produces it.
In fact, I’d say this is also roughly what is going on with the human brain. Some pretty janky mechanisms produce each thought; but the thoughts work together to get some impressive stuff done. Humans would be nothing without system 2, what you call reasoning and don’t address.