The use of something like L1 regularization to achieve sparsity for inherent interpretability may just make things worse; a fixation on L1 regularization may lead people in the wrong direction. To avoid fixation, we should take a step back and look at the big picture. Occam’s razor suggests that we should look for simple (and creative) solutions instead of over-engineering solutions when the entire foundation is inadequate.
In order to obtain inherent interpretability, the machine learning model needs to behave in a way that is interesting to mathematicians. By piling on tweaks such as a lot of L1 or L0 regularization for sparsity, one is making the machine learning model more complicated. That makes it more difficult to study mathematically. And neural networks are inherently difficult to study mathematically and to interpret, so they should be replaced with something else. The problem is that neural networks already have so much momentum that people are unwilling to try anything else, and people are way too indoctrinated into neural networkology that they cannot learn new things.
So how does one get momentum with a non-neural machine learning algorithm? One starts with shallow but mathematical machine learning algorithms first and one can also work with algorithms with few layers too. These shallow/few layer mathematical algorithms can still be effective for some problems since they have plenty of width. One may also construct a hybrid model where the first few layers are the mathematical construction but where the rest of the network is a deep neural network. I do not see how to make a very deep network this way, so the next steps are obscure to me.
The use of something like L1 regularization to achieve sparsity for inherent interpretability may just make things worse; a fixation on L1 regularization may lead people in the wrong direction. To avoid fixation, we should take a step back and look at the big picture. Occam’s razor suggests that we should look for simple (and creative) solutions instead of over-engineering solutions when the entire foundation is inadequate.
In order to obtain inherent interpretability, the machine learning model needs to behave in a way that is interesting to mathematicians. By piling on tweaks such as a lot of L1 or L0 regularization for sparsity, one is making the machine learning model more complicated. That makes it more difficult to study mathematically. And neural networks are inherently difficult to study mathematically and to interpret, so they should be replaced with something else. The problem is that neural networks already have so much momentum that people are unwilling to try anything else, and people are way too indoctrinated into neural networkology that they cannot learn new things.
So how does one get momentum with a non-neural machine learning algorithm? One starts with shallow but mathematical machine learning algorithms first and one can also work with algorithms with few layers too. These shallow/few layer mathematical algorithms can still be effective for some problems since they have plenty of width. One may also construct a hybrid model where the first few layers are the mathematical construction but where the rest of the network is a deep neural network. I do not see how to make a very deep network this way, so the next steps are obscure to me.