If you want to put it that way, nothing conceptually interesting is going on in any neural network paper—we already know they’re universal.
My problem is the details: I visualize neural networks as like pachinko machines or Galton’s quincunx—you drop a bunch of bits (many balls) into the top (bottom) layer of neurons (pins) and they cascade down to the bottom based on the activation functions (spacing of pins & how many balls hit a pin simultaneously), and at the bottom (top) is emitted a final smaller output (1 ball somewhere). I don’t get the details of what it means to add a memory to this many-to-one function.
Recurrent Neural Networks feed their outputs back into their inputs, and run continuously. Instead of visualizing a river flowing directly downstream, visualize a river with a little cove off to one side, where vortices swirl around.
If you want to put it that way, nothing conceptually interesting is going on in any neural network paper—we already know they’re universal.
My problem is the details: I visualize neural networks as like pachinko machines or Galton’s quincunx—you drop a bunch of bits (many balls) into the top (bottom) layer of neurons (pins) and they cascade down to the bottom based on the activation functions (spacing of pins & how many balls hit a pin simultaneously), and at the bottom (top) is emitted a final smaller output (1 ball somewhere). I don’t get the details of what it means to add a memory to this many-to-one function.
Recurrent Neural Networks feed their outputs back into their inputs, and run continuously. Instead of visualizing a river flowing directly downstream, visualize a river with a little cove off to one side, where vortices swirl around.
I think another way to look at neural networks is they are nested non-linear regression models.
I am probably in the minority here, but I don’t think the stuff in the OP is that interesting.
The same way you add memory to many-to-one boolean functions: feedback loops.