We’ve had neural network architectures with a time component for many many years. It’s extremely common. We actually have very sophisticated versions of them that intrinsically incorporate concepts like short-term memory. I wonder if he somehow doesn’t know this, or if he just misspoke, or if I’m misunderstanding what he means.
I assume you’re talking about LSTMs and similar. I think you are misunderstanding what he means. I assumed he was referring to this:
If the neuron is triggered to fire (due to the first type of synapses, the ones near the cell body), and has already been prepared by a dendritic spike, then it fires slightly sooner, which matters because there are fast inhibitory processes, such that if a neuron fires slightly before its neighbors, it can prevent those neighbors from firing at all.
In other words, the analogy here might not be LSTMs, but a multithreaded program where race conditions are critical to its operation :(
EDIT: Maybe not, actually; I missed this part: “This allows networks of neurons to do sophisticated temporal predictions”. My new guess is that he’s referring to predictive processing. I assume self-supervised learning is the analogous concept in ML.
Hmm, it’s true that a traditional RNN can’t imitate the detailed mechanism, but I think it can imitate the overall functionality. (But probably in a computationally inefficient way—multiple time-steps and multiple nodes.) I’m not 100% sure.
I assume you’re talking about LSTMs and similar. I think you are misunderstanding what he means. I assumed he was referring to this:
In other words, the analogy here might not be LSTMs, but a multithreaded program where race conditions are critical to its operation :(
EDIT: Maybe not, actually; I missed this part: “This allows networks of neurons to do sophisticated temporal predictions”. My new guess is that he’s referring to predictive processing. I assume self-supervised learning is the analogous concept in ML.
Hmm, it’s true that a traditional RNN can’t imitate the detailed mechanism, but I think it can imitate the overall functionality. (But probably in a computationally inefficient way—multiple time-steps and multiple nodes.) I’m not 100% sure.