the pioneers’ intuitions might still be misguided, as it seems their initial inclination to work with neural networks was motivated for the wrong reasons: the efficacy of neural networks (probably) comes not from their nominal similarity to biological brains but rather the richness of high-dimensional representations
But they wanted to imitate the brain, because of the brain’s high capabilities. And they discovered neural network architectures with high capabilities. Do you think the brain’s capabilities have nothing to do with the use of high-dimensional representations?
That doesn’t change the fact that the pioneers really only pursued neural networks because of their similarity to the actual structure of the brain, not by first-principles reasoning about how high dimensionality and gradient descent scale well with data, size, and compute (I understand this is a high bar but this is part of why I don’t think there are any real “experts”). And in their early career especially, they were all mired in the neurological paradigm for thinking about neural networks.
Hinton, who got close to breaking free from this way of thinking when he published the backprop paper, ends it by saying “it is worth looking for more biologically plausible ways of doing gradient descent.” In fact, his 2022 forward-forward algorithm shows his approach is still tied to biological plausibility. A 2023 interview with the University of Toronto, he mentions that the reason he got concerned about superintelligence was that when working on the FF algorithm, he realized that backpropagation was just going to be better than any optimization algorithm inspired by the brain.
But they wanted to imitate the brain, because of the brain’s high capabilities. And they discovered neural network architectures with high capabilities. Do you think the brain’s capabilities have nothing to do with the use of high-dimensional representations?
That doesn’t change the fact that the pioneers really only pursued neural networks because of their similarity to the actual structure of the brain, not by first-principles reasoning about how high dimensionality and gradient descent scale well with data, size, and compute (I understand this is a high bar but this is part of why I don’t think there are any real “experts”). And in their early career especially, they were all mired in the neurological paradigm for thinking about neural networks.
Hinton, who got close to breaking free from this way of thinking when he published the backprop paper, ends it by saying “it is worth looking for more biologically plausible ways of doing gradient descent.” In fact, his 2022 forward-forward algorithm shows his approach is still tied to biological plausibility. A 2023 interview with the University of Toronto, he mentions that the reason he got concerned about superintelligence was that when working on the FF algorithm, he realized that backpropagation was just going to be better than any optimization algorithm inspired by the brain.