Yes, that was part of what I was getting at. BNNs cannot implement weight sharing in a direct way like ANNs, in instantiating a bunch of copies in parallel in VRAM; they could do it by a recurrent approach, because what is a RNN but a very wide NN with weight-sharing unrolled sequence-wise? Except then that would require a large number of serial steps—serial steps which would blow any latency budget. So, pace Steve Brynes’s discussion of things like the callosum and where the parameters go in brains, they might have to do pseudo-weight sharing by just replicating a lot of brain regions with similar-ish parameters—and boom, their parameter count spikes massively, and the tighter the latency budget, the worse it gets because the less sharing remains possible through layer-wise or recurrent iteration.
Yes, that was part of what I was getting at. BNNs cannot implement weight sharing in a direct way like ANNs, in instantiating a bunch of copies in parallel in VRAM; they could do it by a recurrent approach, because what is a RNN but a very wide NN with weight-sharing unrolled sequence-wise? Except then that would require a large number of serial steps—serial steps which would blow any latency budget. So, pace Steve Brynes’s discussion of things like the callosum and where the parameters go in brains, they might have to do pseudo-weight sharing by just replicating a lot of brain regions with similar-ish parameters—and boom, their parameter count spikes massively, and the tighter the latency budget, the worse it gets because the less sharing remains possible through layer-wise or recurrent iteration.