I think that one point that the paper doesn’t address is that the model treats “software progress” as basically “how many parallel copies can you run on a fixed device” but the human brain isn’t really parallelizable in that way, and serial compute is expensive (A B300 can do ~15e15 FLOP/s, but it runs at ~2 GHz, so it’s hard-bounded to at most ~2e9 sequential operations per second, and realistically, for AI applications, far fewer). That pushes things slightly more toward being hardware-limited than the compute-constant framing suggests, at least if the sample-efficient architectures turn out to be serial-bound.
One reason current training parallelizes so well is the batch size, an LLM pass processes ~1M tokens at once, so training’s serial depth is tiny relative to its compute. The brain has no such mechanism: it can’t read a million tokens in one shot, it learns from a narrow, time-ordered channel. So if brain-like sample efficiency requires that kind of small-batch learning, a full run would take a large number of sequential steps. However, the brain sets a low bar for speed: a neuron fires ~once a second, so biological learning advances one serial step per second, and transistors should demolish that. But with current hardware you hit a throughput wall, even a sample-efficient model is still big (I’d assume even non-traditional architectures need large memory for real intelligence, although the full memory might be used only for search and not required for a single step), so running it one input at a time is memory-bandwidth-bound, i.e. LLM inference speed. Passing ~1B tokens serially through a 100B-param model is roughly a week at ~1000 tok/s (only slightly better with more GPUs), whereas the same raw FLOP at full utilization is only hours (~1e9 × 100e9 / 1.5e16 ≈ 26,000 s). So the bottleneck flips from compute to serial throughput.
But that ~1000 tok/s is an artifact of one specific operation, which is taking a single activation vector and multiplying it by big parameter matrices. A different (token × parameter) computation, or the same one restructured, could be faster, and current transistor-based GPUs could be used in less conventional ways, or FPGAs could take over if standard GPUs turn out to be fundamentally ill-suited, without requiring new hardware manufacturing.
I imagine that brain-like approaches would require some sort of active learning that is bounded by serial compute/inference, but maybe there could be concurrency or parallelization-based approaches to keep sample efficiency with parallelization, the question is whether those exist even if we allow a superintelligence to design the software.
At 1000 tokens per second, the gap ends up being hours vs. days-or-weeks, so it’s very small compared to the 6-16 OOMs mentioned in the paper, and it shrinks once the hardware is optimized (until cooling/power/transistor efficiency becomes the limiting factor). Also, even if the algorithm is serial-bound, this gap could probably only affect training/learning, or certain tasks.
Overall I think that the idea that sample-efficient architectures might be serial-bound due to having a limited information flow is something to take into account, but there are stacking factors that lower the likelihood of it mattering (new hardware, hardware repurposing for new architectures, high sample efficiency with parallel learning, learning/inference steps not requiring passes through every parameter, and parallel intelligence still being able to perform most tasks). And most importantly, even in the strongest case for serial-inefficiency, it would likely only matter by ~1-2 OOMs.
It’s also worth noting that this specific bottleneck is almost exactly what neuromorphic hardware, and some LLM inference hardware is built to address. The serial inefficiency in the LLM case comes mostly from memory bandwidth, the von Neumann architecture memory-CPU split means weights sit in separate memory and have to be streamed to the compute units every step, and with batch-1 there’s no way to amortize that transfer. Neuromorphic (and more broadly in-memory-compute) architectures attack this directly by putting the memory on the chip itself, co-located with compute, so the weight is physically where it’s used and there’s no fetch, which is precisely the batch-1 case that GPUs are worst at. So there’s already an active hardware research program aimed at the exact wall this argument depends on, which further suggests the serial penalty is soft rather than fundamental.
I think that one point that the paper doesn’t address is that the model treats “software progress” as basically “how many parallel copies can you run on a fixed device” but the human brain isn’t really parallelizable in that way, and serial compute is expensive (A B300 can do ~15e15 FLOP/s, but it runs at ~2 GHz, so it’s hard-bounded to at most ~2e9 sequential operations per second, and realistically, for AI applications, far fewer). That pushes things slightly more toward being hardware-limited than the compute-constant framing suggests, at least if the sample-efficient architectures turn out to be serial-bound.
One reason current training parallelizes so well is the batch size, an LLM pass processes ~1M tokens at once, so training’s serial depth is tiny relative to its compute. The brain has no such mechanism: it can’t read a million tokens in one shot, it learns from a narrow, time-ordered channel. So if brain-like sample efficiency requires that kind of small-batch learning, a full run would take a large number of sequential steps. However, the brain sets a low bar for speed: a neuron fires ~once a second, so biological learning advances one serial step per second, and transistors should demolish that. But with current hardware you hit a throughput wall, even a sample-efficient model is still big (I’d assume even non-traditional architectures need large memory for real intelligence, although the full memory might be used only for search and not required for a single step), so running it one input at a time is memory-bandwidth-bound, i.e. LLM inference speed. Passing ~1B tokens serially through a 100B-param model is roughly a week at ~1000 tok/s (only slightly better with more GPUs), whereas the same raw FLOP at full utilization is only hours (~1e9 × 100e9 / 1.5e16 ≈ 26,000 s). So the bottleneck flips from compute to serial throughput.
But that ~1000 tok/s is an artifact of one specific operation, which is taking a single activation vector and multiplying it by big parameter matrices. A different (token × parameter) computation, or the same one restructured, could be faster, and current transistor-based GPUs could be used in less conventional ways, or FPGAs could take over if standard GPUs turn out to be fundamentally ill-suited, without requiring new hardware manufacturing.
I imagine that brain-like approaches would require some sort of active learning that is bounded by serial compute/inference, but maybe there could be concurrency or parallelization-based approaches to keep sample efficiency with parallelization, the question is whether those exist even if we allow a superintelligence to design the software.
At 1000 tokens per second, the gap ends up being hours vs. days-or-weeks, so it’s very small compared to the 6-16 OOMs mentioned in the paper, and it shrinks once the hardware is optimized (until cooling/power/transistor efficiency becomes the limiting factor). Also, even if the algorithm is serial-bound, this gap could probably only affect training/learning, or certain tasks.
Overall I think that the idea that sample-efficient architectures might be serial-bound due to having a limited information flow is something to take into account, but there are stacking factors that lower the likelihood of it mattering (new hardware, hardware repurposing for new architectures, high sample efficiency with parallel learning, learning/inference steps not requiring passes through every parameter, and parallel intelligence still being able to perform most tasks). And most importantly, even in the strongest case for serial-inefficiency, it would likely only matter by ~1-2 OOMs.
It’s also worth noting that this specific bottleneck is almost exactly what neuromorphic hardware, and some LLM inference hardware is built to address. The serial inefficiency in the LLM case comes mostly from memory bandwidth, the von Neumann architecture memory-CPU split means weights sit in separate memory and have to be streamed to the compute units every step, and with batch-1 there’s no way to amortize that transfer. Neuromorphic (and more broadly in-memory-compute) architectures attack this directly by putting the memory on the chip itself, co-located with compute, so the weight is physically where it’s used and there’s no fetch, which is precisely the batch-1 case that GPUs are worst at. So there’s already an active hardware research program aimed at the exact wall this argument depends on, which further suggests the serial penalty is soft rather than fundamental.