I’d guess that humans have a short term memory of much more than ten bits, though. My intuition is that human short term memory would be much more comparable to neuralese vectors than to single tokens or 4 mental “items”.
But yeah, the conclusions were much more speculative than the model.
And I think insofar as humans are agentic, they apply some general purpose search instead of semi-randomly picking their heuristics? Maybe?
I’d guess that humans have a short term memory of much more than ten bits, though. My intuition is that human short term memory would be much more comparable to neuralese vectors than to single tokens or 4 mental “items”.
Maybe, but I’d guess that’s a difference of less than an order of magnitude—and it seems like the relevant question isn’t only bits passed between circuits, since LLMs, even without reasoning, are autoregressive, so they can reason sequentially over multiple tokens. (And with reasoning, that’s obviously even more true.)
insofar as humans are agentic, they apply some general purpose search instead of semi-randomly picking their heuristics?
To the extent that LLM agents are agents, they definitely do this too! And if we’re talking about single-forward-pass reasoning, very few humans intentionally train their system 1 to do something better than semi-randomly follow patterns that worked before. (If you don’t know what I’m referring to, see the discussion of firefighters not actually making decisions and the resolution of the debate about system 1 / system 2 in Thinking Fast and Slow.)
Yeah, I’m not sure in which ways the analogy works/doesn’t work. (It’s curious that for LLMs, tokens function as observations, as actuators, and as short-term/medium-term memory).
My intuition is still that it’s a large benefit that human can maintain some large-ish latent state while reasoning, while an LLM chain of “thought” could be analogized to a row of copies of the same human standing in sequence, each person hearing what the person before them said, thinking for ¿a second? (100 layers at 5ms for neuronal firing), and then repeating what they heard to their successor, appending a single word. Caveats from sampling methods, e.g. beam search.
When I think about implementing general-purpose search this way my guess is that the bottleneck of squeezing anything into a single most informative token, and my intuition is that general-purpose search needs quite deep serial computations on a world model; or that it’s maybe possible to implement it this way but the token bottleneck is very annoying.
If the token bottleneck is such a problem, then
Staying with readable chains of thought would hinder capabilities progress (so, the more rapid capabilities progress I see, the less I should be confident in the conclusions of this model)
There’d be a strong pressure on finding a way to make latent reasoning scalable
This would be even worse for alignment
If I were to argue against myself, I’d say “each token can add ~10 bits of optimization pressure, but each successive forward pass can add another ~10 bits of optimization pressure to the existing context window, so you end up with a lot of bits of optimization pressure anyway”, then my skeptical answer would be questioning if bits of optimization pressure can be additive, and if so whether that’s the case in this case.
I’d guess that humans have a short term memory of much more than ten bits, though.
LLMs aren’t limited to only tokens as inputs though. They can also attend to internal states as long as they’re in previous layers. This has limits to how much useful data can be passed from previous positions but it’s way more than 10 bits.
But an LLMs’ short-term memory between forward passes includes everything accessible via attention, not just the vertical slice in the current position. Treating the single 10-bit token as the full memory misses the vast majority of the inputs at any given layer.
For example, if an LLM makes a decision in an early layer at position n, it can reference that decision directly in any later layer in positions after n, without going through the tokens.
This is limited since there’s only O(100) layers to work with, but it’s a meaningful amount of memory.
I’d guess that humans have a short term memory of much more than ten bits, though. My intuition is that human short term memory would be much more comparable to neuralese vectors than to single tokens or 4 mental “items”.
But yeah, the conclusions were much more speculative than the model.
And I think insofar as humans are agentic, they apply some general purpose search instead of semi-randomly picking their heuristics? Maybe?
Maybe, but I’d guess that’s a difference of less than an order of magnitude—and it seems like the relevant question isn’t only bits passed between circuits, since LLMs, even without reasoning, are autoregressive, so they can reason sequentially over multiple tokens. (And with reasoning, that’s obviously even more true.)
To the extent that LLM agents are agents, they definitely do this too! And if we’re talking about single-forward-pass reasoning, very few humans intentionally train their system 1 to do something better than semi-randomly follow patterns that worked before. (If you don’t know what I’m referring to, see the discussion of firefighters not actually making decisions and the resolution of the debate about system 1 / system 2 in Thinking Fast and Slow.)
Yeah, I’m not sure in which ways the analogy works/doesn’t work. (It’s curious that for LLMs, tokens function as observations, as actuators, and as short-term/medium-term memory).
My intuition is still that it’s a large benefit that human can maintain some large-ish latent state while reasoning, while an LLM chain of “thought” could be analogized to a row of copies of the same human standing in sequence, each person hearing what the person before them said, thinking for ¿a second? (100 layers at 5ms for neuronal firing), and then repeating what they heard to their successor, appending a single word. Caveats from sampling methods, e.g. beam search.
When I think about implementing general-purpose search this way my guess is that the bottleneck of squeezing anything into a single most informative token, and my intuition is that general-purpose search needs quite deep serial computations on a world model; or that it’s maybe possible to implement it this way but the token bottleneck is very annoying.
If the token bottleneck is such a problem, then
Staying with readable chains of thought would hinder capabilities progress (so, the more rapid capabilities progress I see, the less I should be confident in the conclusions of this model)
There’d be a strong pressure on finding a way to make latent reasoning scalable
This would be even worse for alignment
If I were to argue against myself, I’d say “each token can add ~10 bits of optimization pressure, but each successive forward pass can add another ~10 bits of optimization pressure to the existing context window, so you end up with a lot of bits of optimization pressure anyway”, then my skeptical answer would be questioning if bits of optimization pressure can be additive, and if so whether that’s the case in this case.
LLMs aren’t limited to only tokens as inputs though. They can also attend to internal states as long as they’re in previous layers. This has limits to how much useful data can be passed from previous positions but it’s way more than 10 bits.
I meant between forward passes. Not within a forward pass.
But an LLMs’ short-term memory between forward passes includes everything accessible via attention, not just the vertical slice in the current position. Treating the single 10-bit token as the full memory misses the vast majority of the inputs at any given layer.
For example, if an LLM makes a decision in an early layer at position n, it can reference that decision directly in any later layer in positions after n, without going through the tokens.
This is limited since there’s only O(100) layers to work with, but it’s a meaningful amount of memory.