Interesting idea, and I’m generally very in favour of any efforts to find more understandable and meaningful “elementary units” of neural networks right now. I think this is currently the research question that most bottlenecks any efforts to get a deeper understanding of NN internals and NN selection, and I think those things are currently the biggest bottlenecks to any efforts at generating alignment strategies that might actually work. So we should be experimenting with lots of ideas for different NN “bases” to use and construct our theory of Deep Learning on top of, until we get a strong signal that we’ve found the right one.
Both bases that keep the layer structure the same, such as the one you propose here, or the one we’re planning to investigate next, and bases that assume the layer structure doesn’t quite match the way we should be thinking about the time ordering of computations in the network, and allow basis transformations that put activations that used to be in different layers into the same layer.
If anyone is looking to come up with more promising ideas for basis transformations, some guiding heuristics to generate candidates might be: bases that seem to spontaneously show up when you’re investigating the math behind some property of neural networks, bases that seem to make neural networks a lot more understandable to humans without requiring a lot of effort to compute, bases that come out of some theory or hypothesis of what neural networks are “really doing”, bases that have less degrees of freedom than the neuron basis but still seem to accurately capture the behaviour of the network in many aspects of training and deployment both.
Interesting idea, and I’m generally very in favour of any efforts to find more understandable and meaningful “elementary units” of neural networks right now. I think this is currently the research question that most bottlenecks any efforts to get a deeper understanding of NN internals and NN selection, and I think those things are currently the biggest bottlenecks to any efforts at generating alignment strategies that might actually work. So we should be experimenting with lots of ideas for different NN “bases” to use and construct our theory of Deep Learning on top of, until we get a strong signal that we’ve found the right one.
Both bases that keep the layer structure the same, such as the one you propose here, or the one we’re planning to investigate next, and bases that assume the layer structure doesn’t quite match the way we should be thinking about the time ordering of computations in the network, and allow basis transformations that put activations that used to be in different layers into the same layer.
If anyone is looking to come up with more promising ideas for basis transformations, some guiding heuristics to generate candidates might be: bases that seem to spontaneously show up when you’re investigating the math behind some property of neural networks, bases that seem to make neural networks a lot more understandable to humans without requiring a lot of effort to compute, bases that come out of some theory or hypothesis of what neural networks are “really doing”, bases that have less degrees of freedom than the neuron basis but still seem to accurately capture the behaviour of the network in many aspects of training and deployment both.
I agree entirely with this bottleneck analysis, and am also very excited about the work you’re doing and have just posted.