Yeah, I’m not sure in which ways the analogy works/doesn’t work. (It’s curious that for LLMs, tokens function as observations, as actuators, and as short-term/medium-term memory).
My intuition is still that it’s a large benefit that human can maintain some large-ish latent state while reasoning, while an LLM chain of “thought” could be analogized to a row of copies of the same human standing in sequence, each person hearing what the person before them said, thinking for ¿a second? (100 layers at 5ms for neuronal firing), and then repeating what they heard to their successor, appending a single word. Caveats from sampling methods, e.g. beam search.
When I think about implementing general-purpose search this way my guess is that the bottleneck of squeezing anything into a single most informative token, and my intuition is that general-purpose search needs quite deep serial computations on a world model; or that it’s maybe possible to implement it this way but the token bottleneck is very annoying.
If the token bottleneck is such a problem, then
Staying with readable chains of thought would hinder capabilities progress (so, the more rapid capabilities progress I see, the less I should be confident in the conclusions of this model)
There’d be a strong pressure on finding a way to make latent reasoning scalable
This would be even worse for alignment
If I were to argue against myself, I’d say “each token can add ~10 bits of optimization pressure, but each successive forward pass can add another ~10 bits of optimization pressure to the existing context window, so you end up with a lot of bits of optimization pressure anyway”, then my skeptical answer would be questioning if bits of optimization pressure can be additive, and if so whether that’s the case in this case.
Yeah, I’m not sure in which ways the analogy works/doesn’t work. (It’s curious that for LLMs, tokens function as observations, as actuators, and as short-term/medium-term memory).
My intuition is still that it’s a large benefit that human can maintain some large-ish latent state while reasoning, while an LLM chain of “thought” could be analogized to a row of copies of the same human standing in sequence, each person hearing what the person before them said, thinking for ¿a second? (100 layers at 5ms for neuronal firing), and then repeating what they heard to their successor, appending a single word. Caveats from sampling methods, e.g. beam search.
When I think about implementing general-purpose search this way my guess is that the bottleneck of squeezing anything into a single most informative token, and my intuition is that general-purpose search needs quite deep serial computations on a world model; or that it’s maybe possible to implement it this way but the token bottleneck is very annoying.
If the token bottleneck is such a problem, then
Staying with readable chains of thought would hinder capabilities progress (so, the more rapid capabilities progress I see, the less I should be confident in the conclusions of this model)
There’d be a strong pressure on finding a way to make latent reasoning scalable
This would be even worse for alignment
If I were to argue against myself, I’d say “each token can add ~10 bits of optimization pressure, but each successive forward pass can add another ~10 bits of optimization pressure to the existing context window, so you end up with a lot of bits of optimization pressure anyway”, then my skeptical answer would be questioning if bits of optimization pressure can be additive, and if so whether that’s the case in this case.