human-level human-speed AGI will require not a data center, but rather something like one consumer gaming GPU—and not just for inference, but even for training from scratch.
A bit of a necrocomment, but I really don’t think this is correct.
Human brain is estimated to have 100 trillion synaptic connections, which we can roughly, very roughly, consider equivalent to neural network parameters. We’ll poke this assumption more later. For now let’s assume we’ll need a 100T parameter model to match or surpass human cognition. At FP8, that’s 100 trillion bytes. That’s 100 terabytes of VRAM needed to run it, and that is even before KV-cache, activations and any other stuff (such as Adam optimizer states). No single consumer GPU on the planet currently has that much VRAM, not even within the same order of magnitude. Going to FP4 or FP2 (if FP2 training is even possible, which I frankly doubt) would only cut that number by 2x and 4x, respectively.
What if to match 100 trillion synaptic connections you only need 10 trillion parameters in a neural network? Then you’ll need 10 terabytes of VRAM to run it. 5 terabytes at FP4 just for the weights alone. Still nowhere near what a consumer GPU can run.
Ok, let’s make maximally generous assumptions. Let’s say you only need 1 trillion parameters to match a human brain (100 times less than the naïve 100T number based on synaptic connections), and let’s say FP2 training is magically possible. Let’s say this neural net is not a Transformer, so no KV-cache (well, KV-cache isn’t used during training anyway, only during inference), and let’s say there is no extra overhead from anything such as activations/Adam optimizer states/gradients. That would still be around 250 GB of VRAM. RTX 5090 has 32 GB of VRAM. If you stretch the definition of “consumer gaming GPU” really far, H200 has 141 GB of VRAM, which is still not enough, but at least it’s within the right order of magnitude.
So no, brain-like AGI will require significantly more advanced hardware than present-day consumer-grade GPUs in the absence of a neural network architecture many orders of magnitude more efficient than what the naïve comparison suggests.
A bit of a necrocomment, but I really don’t think this is correct.
Human brain is estimated to have 100 trillion synaptic connections, which we can roughly, very roughly, consider equivalent to neural network parameters. We’ll poke this assumption more later. For now let’s assume we’ll need a 100T parameter model to match or surpass human cognition. At FP8, that’s 100 trillion bytes. That’s 100 terabytes of VRAM needed to run it, and that is even before KV-cache, activations and any other stuff (such as Adam optimizer states). No single consumer GPU on the planet currently has that much VRAM, not even within the same order of magnitude. Going to FP4 or FP2 (if FP2 training is even possible, which I frankly doubt) would only cut that number by 2x and 4x, respectively.
What if to match 100 trillion synaptic connections you only need 10 trillion parameters in a neural network? Then you’ll need 10 terabytes of VRAM to run it. 5 terabytes at FP4 just for the weights alone. Still nowhere near what a consumer GPU can run.
Ok, let’s make maximally generous assumptions. Let’s say you only need 1 trillion parameters to match a human brain (100 times less than the naïve 100T number based on synaptic connections), and let’s say FP2 training is magically possible. Let’s say this neural net is not a Transformer, so no KV-cache (well, KV-cache isn’t used during training anyway, only during inference), and let’s say there is no extra overhead from anything such as activations/Adam optimizer states/gradients. That would still be around 250 GB of VRAM. RTX 5090 has 32 GB of VRAM. If you stretch the definition of “consumer gaming GPU” really far, H200 has 141 GB of VRAM, which is still not enough, but at least it’s within the right order of magnitude.
So no, brain-like AGI will require significantly more advanced hardware than present-day consumer-grade GPUs in the absence of a neural network architecture many orders of magnitude more efficient than what the naïve comparison suggests.