I agree, a one-trillion parameter model (~4Tb) can probably simulate the brain well enough, and you might be able to squeeze out another order or two of magnitude. The brain only has a hundred billion neurons after all, and maybe the reason it needs many more synapses is because its clock speed is a hundred million times slower than a GPU.
ETA: Also, only ~1% of human genes are coding, so you arguably only need 10 Mb for the evolved brain.
What it takes to simulate the brain is not really relevant here. What matters is how many bits you currently have access to. If something is not part of your direct consciousness, you don’t need to specify it in the program. This includes memories you are not currently thinking about.
I think we’re looking at this from two different directions. Mine is, once trained, how many bits are in the structure of the brain? Yours is, during training, how many bits does the brain absorb? The numbers should come out the same, but mine is easier to get a lower bound and yours is easier to get an upper bound with. You have to worry about issues like, “what about memory that isn’t being accessed in this compute cycle?” while I made the assumption that every parameter is contributing 1 bit of information to the active computation.
My assumption is actually too strong though. Mixture of experts poses a problem, but a bigger problem is that it actually takes many parameters per bit of information. Probably. I mean, you only need to add a few hundred bits to train MNIST to >95% accuracy, while it takes ~10k CNN parameters.
(note: this is trained on a 300k ResNet, not a CNN).
I agree, a one-trillion parameter model (~4Tb) can probably simulate the brain well enough, and you might be able to squeeze out another order or two of magnitude. The brain only has a hundred billion neurons after all, and maybe the reason it needs many more synapses is because its clock speed is a hundred million times slower than a GPU.
ETA: Also, only ~1% of human genes are coding, so you arguably only need 10 Mb for the evolved brain.
What it takes to simulate the brain is not really relevant here. What matters is how many bits you currently have access to. If something is not part of your direct consciousness, you don’t need to specify it in the program. This includes memories you are not currently thinking about.
I think we’re looking at this from two different directions. Mine is, once trained, how many bits are in the structure of the brain? Yours is, during training, how many bits does the brain absorb? The numbers should come out the same, but mine is easier to get a lower bound and yours is easier to get an upper bound with. You have to worry about issues like, “what about memory that isn’t being accessed in this compute cycle?” while I made the assumption that every parameter is contributing 1 bit of information to the active computation.
My assumption is actually too strong though. Mixture of experts poses a problem, but a bigger problem is that it actually takes many parameters per bit of information. Probably. I mean, you only need to add a few hundred bits to train MNIST to >95% accuracy, while it takes ~10k CNN parameters.
(note: this is trained on a 300k ResNet, not a CNN).