Feels like there’s a missing deep learning paradigm that is the equivalent of the human “go for a walk and think about stuff, absent new stimuli”. There are some existing approaches that hint at this (generative replay / dreaming) but those feel a bit different than my subjective sense that I’m “working things out” when I go for a walk, rather than generatively dreaming, as I do at night.
Relatedly: it reduces my overall cognitive output when I go through periods of depriving myself of these idle periods by jamming them full of stimuli (e.g. podcasts). I do better when I bite the bullet, accept the boredom, and allow my mind to work out whatever it needs to.
Speculatively, perhaps the two are related: overstimulated ML researchers have a blind spot to the fact that understimulating a neural net might be helpful.
Feels like there’s a missing deep learning paradigm that is the equivalent of the human “go for a walk and think about stuff, absent new stimuli”. There are some existing approaches that hint at this (generative replay / dreaming) but those feel a bit different than my subjective sense that I’m “working things out” when I go for a walk, rather than generatively dreaming, as I do at night.
Relatedly: it reduces my overall cognitive output when I go through periods of depriving myself of these idle periods by jamming them full of stimuli (e.g. podcasts). I do better when I bite the bullet, accept the boredom, and allow my mind to work out whatever it needs to.
Speculatively, perhaps the two are related: overstimulated ML researchers have a blind spot to the fact that understimulating a neural net might be helpful.