What it takes to simulate the brain is not really relevant here. What matters is how many bits you currently have access to. If something is not part of your direct consciousness, you don’t need to specify it in the program. This includes memories you are not currently thinking about.
I think we’re looking at this from two different directions. Mine is, once trained, how many bits are in the structure of the brain? Yours is, during training, how many bits does the brain absorb? The numbers should come out the same, but mine is easier to get a lower bound and yours is easier to get an upper bound with. You have to worry about issues like, “what about memory that isn’t being accessed in this compute cycle?” while I made the assumption that every parameter is contributing 1 bit of information to the active computation.
My assumption is actually too strong though. Mixture of experts poses a problem, but a bigger problem is that it actually takes many parameters per bit of information. Probably. I mean, you only need to add a few hundred bits to train MNIST to >95% accuracy, while it takes ~10k CNN parameters.
(note: this is trained on a 300k ResNet, not a CNN).
What it takes to simulate the brain is not really relevant here. What matters is how many bits you currently have access to. If something is not part of your direct consciousness, you don’t need to specify it in the program. This includes memories you are not currently thinking about.
I think we’re looking at this from two different directions. Mine is, once trained, how many bits are in the structure of the brain? Yours is, during training, how many bits does the brain absorb? The numbers should come out the same, but mine is easier to get a lower bound and yours is easier to get an upper bound with. You have to worry about issues like, “what about memory that isn’t being accessed in this compute cycle?” while I made the assumption that every parameter is contributing 1 bit of information to the active computation.
My assumption is actually too strong though. Mixture of experts poses a problem, but a bigger problem is that it actually takes many parameters per bit of information. Probably. I mean, you only need to add a few hundred bits to train MNIST to >95% accuracy, while it takes ~10k CNN parameters.
(note: this is trained on a 300k ResNet, not a CNN).