I think I’m sympathetic to your overall point. That said, I am less pessimistic than you that Neural Network computation can never be understood beyond the macroscopic level ′ what does it do’ .
The Turing machine paradigm is just one out of many paradigms to understand computation. It would be a mistake to be too pessimistic based on just the failure of the ur-classical TM paradigm.
Computational learning theory’s bounds are vacuous for realistic machine learning. I would guess, and I say this as a nonexpert, that this is chiefly due to
(i) a general immaturity of the field of computational complexity, i.e. most of the field is conjectures, it’s hard to prove much about time complexity even if we’re quite confident the statements are likely true
(ii) computational learning theory grew out of classical learning theory and has not fully incorporated the lessons of singular learning theory. Much of the field is working in the wrong ‘worst-case/pessimistic’ framework when they should be thinking in terms of Bayesian inference & simplicity/degeneracy bias. Additionally, there is perhaps too much focus on exact discrete bounds when instead one should be thinking in terms of smooth relaxation and geometry of loss landscapes.
That said, I agree with you that the big questions are currently largely open.
Nice post Cole.
I think I’m sympathetic to your overall point. That said, I am less pessimistic than you that Neural Network computation can never be understood beyond the macroscopic level ′ what does it do’ .
The Turing machine paradigm is just one out of many paradigms to understand computation. It would be a mistake to be too pessimistic based on just the failure of the ur-classical TM paradigm.
Computational learning theory’s bounds are vacuous for realistic machine learning. I would guess, and I say this as a nonexpert, that this is chiefly due to
(i) a general immaturity of the field of computational complexity, i.e. most of the field is conjectures, it’s hard to prove much about time complexity even if we’re quite confident the statements are likely true
(ii) computational learning theory grew out of classical learning theory and has not fully incorporated the lessons of singular learning theory. Much of the field is working in the wrong ‘worst-case/pessimistic’ framework when they should be thinking in terms of Bayesian inference & simplicity/degeneracy bias. Additionally, there is perhaps too much focus on exact discrete bounds when instead one should be thinking in terms of smooth relaxation and geometry of loss landscapes.
That said, I agree with you that the big questions are currently largely open.