From what i’ve seen even the larger synapses store only about 5 bits ish, and the ‘median’ or typical synapse probably stores less than 1 bit in some sense (as the typical brain synapse only barely exists in a probabilistic sense—as in a neuromorphic computer a physical synaptic connection is an obvious but unappreciated prerequisite for a logical synapse, but the former does not necessarily entail the latter: see also quantal synaptic failure).
In my 2022 roadmap I estimated brain capacity at 1e15 bits but that’s probably an overestimate for logical bits.
Also the brain is quite sparse for energy efficiency, but that usually comes at a tradeoff in parameter efficiency. This is well expored in the various tradeoffs for ANNS that model 3d space (NERFs etc) but generalizes to other modalities. The most parameter efficient models will be more dense but less compute/energy efficient for inference as a result. There are always more ways to compress the information stored in an ANN, but those optimization directions are extremely unlikely to align with the optimizations favoring more efficient inference via runtime sparsity (and extreme runtime sparsity probably requires redundancy aka anti-compression).
From what i’ve seen even the larger synapses store only about 5 bits ish, and the ‘median’ or typical synapse probably stores less than 1 bit in some sense (as the typical brain synapse only barely exists in a probabilistic sense—as in a neuromorphic computer a physical synaptic connection is an obvious but unappreciated prerequisite for a logical synapse, but the former does not necessarily entail the latter: see also quantal synaptic failure).
In my 2022 roadmap I estimated brain capacity at 1e15 bits but that’s probably an overestimate for logical bits.
Also the brain is quite sparse for energy efficiency, but that usually comes at a tradeoff in parameter efficiency. This is well expored in the various tradeoffs for ANNS that model 3d space (NERFs etc) but generalizes to other modalities. The most parameter efficient models will be more dense but less compute/energy efficient for inference as a result. There are always more ways to compress the information stored in an ANN, but those optimization directions are extremely unlikely to align with the optimizations favoring more efficient inference via runtime sparsity (and extreme runtime sparsity probably requires redundancy aka anti-compression).